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SUMMARY 


Approach 

A study  was  conducted  to  investigate  and  statistically 
validate  a performance  measuring  system  in  the  Tactical  Air 
Command  Air  Combat  Engagement  Simulator  I air  combat  maneu- 
vering (ACM)  training  program  at  Vought.  Corporation,  Dallas,' 
Texas.  The  study  utilized  a 12  week  sample  of  >39  student 
pilots  in  an  experiment  to  statistically  validate,  an  objec- 
tive performance  measure  of  air  combat  skill,  compare  the 

objective  measure  to  the  subjective  judgement  of  ACM  skill 

■*.*»»  • 

made  by  instructor  pilots,  to  investigate  improvements  as  a 
measure  of  ACM  skill',  and  to  evaluate  its  utility  as  a train- 
ing aid. 

Statistical  methodologies  of  ridge  regression  and  dis- 
criminant analyses  were  employed  to  assess  the  quantitative 
and  qualitative  characteristics  of  the  measure  of  ACM  skill 
in  the  simulator. 

Background 

A scoring  system  termed  the  Good  Stick  Index  (GSI)  is 
used  as  an  indicator  of  pilot,  air  combat  skill  in  the  TAC 
ACES  I simulator  training  program.  The  GSI  was  developed 
jointly  by  the  Tactical  Air  Command  and  the  Vought  Corpora- 
tion utilizing  four  subjectively  chosen  and  equally  weighted 
parameters  which  to  the  experienced’  pilot  are  intuitive  in- 
dicators of  air  combat  skill.  The  four  parameters  are  1) 
time  in  gun  firing  envelope,  2)  average  mil  error,  3)  offen- 
sive/defensive time,  and  4)  time  to  first  kill  - objective 
measures  obtained  during  student  pilot  scoring  sessions 
against  programmed  target  maneuvers.  The  TAC  ACES  I 
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training  program  is  concluded  by  a one-on-one  free  engage- 
ment tournament  where  one  student  pilot  is  matched1  against 
another.  The  turkey  shoot  tournament  is  a double  elimina- 
tion event  (pilots  must  lose  two  engagements  to  be  elimina- 
ted) reisulting  in  a single  winner. 

The  GSI  score,  applied  as  a predictor  of  turkey  shoot 
placement,  appeared  to  predict  the  winner  at  greater  than 
random  frequency. 

Specifics 

The  GSI  validation  study  was  conducted  to  statistically 
validate  the  GSI  as<  a predictor  of  turkey  shoot  winner,,  in- 
vestigate improvement  in  the  GSI  by  varying  the  weighting 
of  each  of  the  four  parameters,  and  introducing  additional 
parameters  as  candidates  for  an  improved  predictor  of  turkey 
shoot  winner.  The  validated  GSI  was  compared  with  the  tur- 
key shoot  student  placement  predictions  of  the  instructor 
pilot  to  assess  its  agreement  with  expert  opinion. 

In  order  to  better  evaluate  the  potential  utility  of 
the  GSI,  four  groupings  of  turkey  shoot  placements  in  each 
class  of  eight  students  were  investigated; 

1)  Winners 

2)  Winners  and  Runners-Up  (Finalists). 

3)  Upper-Half  (Semi-Finalists) 

4)  Quartile  Rankings, 

Data  used  in  the  study  were  collected  during  the  12 
class  (12  week)  sample  from  3 April  1978  through  23  June 
1978.  These  data  were  objective  measures  of  performance 
in  the  simulator,  demographic  (background)  data  obtained 
by  student  questionnaire,  and  instructor  pilots*  predic- 
tions of  turkey  shoot  placement  of  students  within  each 
class.  The  objective  measures  were  obtained  from  scoring 
sessions  on  Mondays,  .immediately  after  briefing  and  hands-on 
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familiarization,  and  on'  Fridays,  just  prior  to  the  turkey 
shoot  exorcise.  In  four  of  the  12  classes,  an  addi- 
tional scoring  session  was  held  on  Wednesdays  to  better 
assess  learning  .trends  in  the  simulator.. 

The  TAC  ACES  I training-  syllabus  was  consistent 
throughout  the  experiment  as  attested  to  by  the  Chief  In-r 
s true tor  Pilots.  Instructor  pilots  provided  individual 
instruction  to  each  student,  concentrating  in  areas  of 
recognized  deficiencies..  The,  students  were  aware  of  ijhe 
scoring  sessions,  but  were  unaware  of  the  intended  use  of 
the  acquired  data.  - 

Results 

The  first  statistical  analysis  performed  determined 
the  prediction  capability  of  the?  equally  weighted,  four 
parameter  GSI  score  obtained  in  Friday  scoring  sessions. 

Thev  results,  were  compared  to  the  subjective  student  turkey 
shoot  rank  predictions  of  the  instructor  pilots . The  anali  i - 
sis  showed  the  GSI  score,  using  Friday  only  data,  to -predict 
the  turkey  shoot  winner  with  a 25  percent  probability  «.oi.e 
in  four)  . There  was  no  statistical  difference  between  vVie 
GSI  and  the  instructor  pilot  prediction  capabilities. 

A second  analysis  summed  the  GSI  score  obtained  on 
Friday  to  the  GSI  score  obtained  oh  Monday  and  optimally 
weighted  the  combined  score.  A significant  increase  in 
.probability  of  correct  turkey  shoot  placement  was  observed 
at  about  66  percent  (two  in  three) . 

A third  analysis  used  the  four  individual  parameters 
of  each  GSI  score  for  Monday  and  Friday  (a  total  of  eight 
terms)  and  optimally  weighted  each  individual  parameter. 

The  results  increased  the  prediction  of  turkey  shoot  place- 
ment to  about  75  percent  (three  in  four) , the  best  prediction 
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which,  could  be  obtained  with  the  four  parameters  intui- 
tively chosen  as  indicators  of  ACM  skill, 

Irt  the  fourth  statistical  analysis,  a set  of  40  objec- 
tive measures  taken  during  each  scoring  session  were  in- 
troduced to  die  discriminant  model  as  potential  predictor 
candidates.  Included  in  the  d'aca  set  were  the  four  para- 
meters in  the  original:  and.  improved  GSI  score.  The  analysis 
derived  an  optimal  predictor  with  about  80  percent  probabi- 
lity of  correct  turkey  shoot  placement.  Further,  a set  of 
12  Subjectively  chosen  demographic  (background)  data  obtained 
from  student  questionnaires  introduced,  as  potential  contribu- 
tor candidates  in  the  expanded;  list  of.  candidates.  The  pro- 
bability of  correct  turkey  sheer,  placement  remained  about 
80  percent  — however,  background  parameters  of  total  time 
in  fighter  aircraft,  time  in  the  F-4  aircraft,  and  the  num- 
ber of  sorties  flown  in  the  last  thirty  days,  replaced  three 
of  the  terms1  in  the  optimal  objective  predictor  score.  This 
result  reinforces  the  predictor  model  as  a measure  of  pilot 
ACM  skill,  1 

The  statistically  validated  GSI  was  used  in  the  final 
analysis  to  obtain  a measure  o'f  learning  trends  in  the  simu- 
lator. A third  scoring  session  on  Wednesday,  in  addition  to 
the  Monday  and  Friday  data,  enabled  an  evaluation  of  skill 
development  in  the  simulator  over  the  week's  training  period. 
A quadratic  fit  through  the  means  of  individual  scores  ob- 
tained on  the  three  days  showed  definite  positive  group 
learning  (edumetric  trend) i The  distribution  of  individual 
scores  was  seen  to  converge,  or  group  closer  together, 
from  Monday  to  Friday.  The  slope  of  the  quadratic  fit 
approached  zero  on  Friday,  which  indicates  that  one  week's 
training  in  the  simulator  was  optimal  for  the  Classes  sub- 
jected to  the  investigation. 


Conclusions 


-The  overall  analyses  in  the  study  showed;  the  GSI  to  be 
■'a  measure  of  ,ACM'  skill  with  contributing  parameters ^cohsis- 
tent  with  intuitive  expert  opinion  and  with  an  .acceptable 
level  of  accurate  assessment  of  skill  in  the  simulator.  The 
GSI  score  is  shown  to  be  useful  in  evaluating  individual 
and  group  learning  within  training  programs  in  ACM,  and  the 
individual  .parameters  comprising  the  GSI  score  can  be  used 
_ as  teaching  guides . 

A recommendation  is  made  to  utilize  the  algorithms  and 
similar  techniques  and  methodologies  as  presented  in  this 
study  to  derive  performance  measurement  systems  for  the 
Simulator  for  Air-to-Air  Combat  at  Luke  AFB  and  the  Air 
Combat  Maneuvering  Instrumentation  (ACMI)  Range  at  Nellis  AFB 
When  an  objective  performance  measure  can  be  obtained  for 
ACM  in  the  air,,  then  an  objective  measure  of  transfer  of 
training  between  the  simulator  and  the  aircraft  can  be  as- 
certained; 

Applications  of  the  techniques  of  the  study  can  also  be 
applied  to  other  ACM  simulators  and  other  types  of  flight 
simulators  to  achieve  like  measures  of  skill  in  a variety 
of  flying  tasks. 


PREFACE 


This  report  documents  the  -tasks  performed  under  con- 
tract F34601-77-A-0176tKW01,  the  Good  Stick  Index  Validation 
Study.  The  Vought  Corporation,  Dallas,  Texas,  has  been 
under  contract  with  the  USAF  Tactical  Air  Command  (TAG)  to 
furnish  the  Air  Combat  Engagement  Simulator  (ACES)  facility 
in  support,  of  TAC  air  combat  training,  during  the  data  collec- 
tion phase  of  this  study.  A pilot  performance  scoring  sys- 
tem, the  Good  Stick  Index  (GSI) , was  developed  earlier  for 
the  purpose  of  predicting  relative  performance,  of  student, 
pilots  in  a free  engagement  competition  within  each  class  of 
eight  pilots.  Initially,  four  parameters  of  pilot  perfor- 
mance were  used  to  compute  a GSI  score,  lor  each  pilot. 

These  parameters  were  selected  subjectively  and  were  em- 
pirically weighted  in  the  scoring  equation.  There  had  been 
no  previous  effort  to  statistically  validate  the.  predictive 
ability  of  the  GSI  equation. 

The  contractor  wishes  to  acknowledge  the  technical 
guidance  ana,  assistance  provided  by  Mr.  Robert  E.  Coward;, 
Contract  Manager  and  Co-Author,  Flying  Training  Division  of 
the  Air  Force  Human  Resources  Laboratory,  and  the  program 
training,  planning,  and  scheduling  interface  of  TAC  ACES  I 
personnel  provided  by  Lt„  Col.  John  K.  Sloan  II  of  the  Air 
Force  Tactical  Fighter  Weapons  Center. 
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I . INTRODUCTION 


The  Good  Stick  Index  (GSI)  is  a numerical  index  devel- 
oped to  measure  student  pilot  proficiencies  in  simulated 
one-on-one  air  combat.  The  GSI,  as  originally  formulated  by 
the  Vought  Corporation,  Dallas,  Texas,  consists  of  four 
objective  performance  parameters,  measured  during  USAF  Tacti- 
cal Air  Command  (TAC)  Air  Combat  Engagement  Simulator  (ACES) 

I training. 

The  four  parameters  comprising  the  GSI  were  subjec- 
tively chosen  and,  from  data  obtained  over  many  classes, 
empirically  related  to  derive  a predictor  of  the  "winner" 
or  "runner-up"  in  the  double  elimination  one-on-one  free 
engagement  tournament  held  at  the  conclusion  of  each  train- 
ing session.  This  derived  relationship  appears  to  predict 
the  winner  or  runner-up  of  the  double  elimination  free  engage- 
ment "turkey-shoot"  with  greater  than  random  frequency. 

This  study  investigates  the  predictive  ability  of  the 
empirically  derived  relationship  as  a predictor  of  turkey 
shoot  winner  by  utilizing  statistical  analysis  methods. 
Further,  the  study  derives,  through  statistical  techniques, 
the  optimal  predictor  indices  using  the  original  four  sub- 
jectively chosen  parameters  and  then  derives  optimal  pre- 
dictors from  an  expanded  set  of  objective  measures,  which 
include  the  four  parameters  originally  chosen. 

These  analyses  were  performed  using  data  collected  from 
12  classes  of  students  in  an  experiment  representative  of 
TAC  ACES  I training.  Input  data  fidelity  was  assured  by 
(a)  certification  that  there  was  adherence  to  the  training 
syllabus  by  the  Instructor  Pilots  (IPs) , (b)  certification 
that  there  were  no  hardware  anomalies,  and  (c)  certifica- 
tion that  there  were  ho  software  anomalies  unaccounted  for 
during  the  control  period. 
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Additional  analyses  were  performed  to  obtain  correla- 
tions of  student  , pilot  background  data  and  IP  subjective 
predictions  of  student  ranking  relative  to  GSI  scores  and 
actual  turkey  shoot  rankings. 

Four  of  the  12  classes  in  the  experiment  were  struc- 
tured to  collect  additional  edumetric  and  psychometric 
parameters  in  order  to  obtain  a greater  measure  of  indivi- 
dual and  group  transfer  of  training  in  the  simulator. 

The  optimal  GSI  predictors,  as  derived  by  statistical 
analyses  of  the  experiment  data,  are  evaluated  as  a predic- 
tor. Using  previous,  class  sessions  as  a data  base  to  a 
limited  degree,  an  assessment  is  made  of  actual  turkey  shoot 
prediction  capability. 

BACKGROUND 

The  TAC  ACES  I training  program  is  conducted  by  the 
Tactical  Air  Command  using  the  Vought  Corporation  fixed 
base  air  combat  simulator  (Figure  1) . The  program  utilizes 
two  F-4  configured  cockpits  with  full  instruments  and  wea- 
pon systems  indicators  necessary  for  air-to-air  combat 
simulation  in  a functional  mode*  The  software  modeling 
is  for  F-4D  and  F-4E  aircraft  flight  characteristic.  In 
addition,  a MIG  21  is  modeled  to  provide  training  in  dis- 
similar aircraft  engagements. 

Facility  Description 

The  Vought  Air  Combat  Simulator,  Figure  1,  consists  of 
two  cockpits,  each  situated  within  16-foot-diameter  spheri- 
cal screens.  Overhead  projectors  provide  dynamic  earth/sky 
horizon  scenes  and  an  image  of  the  opponent's  aircraft. 

The  aircraft  target  is  a high- resolution  color  image  pro- 
vided by  the  Opaque  Target  Optical  Project  System  (OTOPS) , 
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.Figure  1,  The  Vought  Air  Combat  Simulator 

recently  developed  by  Vought.  Each  pilot  wears  a g-suit 
and  sits  on  a g-seat.  As  a -pilot  increases  the  load  fa.otor 
on  his  aircraft,  his  g-suit  inflates  and  his  g-seat  de- 
flates. The  visual  display  dims  as  a function  of  g and 
time  and  finally  blacks  out,  with  the  target  image  the  last 
to  go.  The  g-seat  also  provides  a buffet  cue,  beginning 
as  a high  frequency  nibble,  increasing  in  amplitude  and 
decreasing  in  frequency  as  penetration  into  the  buffet  area 
occurs.  Each  cockpit  is  equipped  with  fire  control  switch- 
ology  which  reflects  the  f-4e,  number  556  and  subsequent, 
as  modified  by  T.O.  1F-4E-556. 
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On-line  firing  and  hit  cues,  engine,  aircraft,  and  wea- 
pon sounds  add  to  the  realism  of  the  simulated  air  combat,  and 

a separate  bullet  model  includes  the  time  of  flight.  Wea- 
pon realism  extends  to  the  heat  and  radar  missiles,  too, 
as  a miss  will  be  scored  if  the  aircraft  target  exceeds  the 
missile  turning/tracking  capabilities  before  the  time,  of 
flight  has  elapsed.  A pilot  scoring  system  called  the  GSI 
measures  the  relative  air  combat  skills  of  the  pilot, 

A unique  Instructor  Pilot  (IP)  station  that  is  mobile 
and  that  can  be  operated  from  alongside  the  cockpit  pro- 
vides the  IP  a matchless  vantage  point.  The  IP  station 
provides  complete  control  of  the  simulation,  including 
operate,  freeze  or  reset,  replay,  data  recording,  video 
recording,  and  options  to  record  and  play  back  preprogrammed 
or  canned  target  trajectories.  It  also  contains  the  engage- 
ment scene  which  can  be  recorded  on  video  cassettes,  along 
with  the  audio  from  both  cockpits  and  the  IP,  for  subsequent 
replay  and  debriefing. 

Training  Sessions 


Typically,  the  TAC  ACES  I training  session  is  sche- 
duled for  one  week  and  consists  of  eight  student  pilots  and 
three  IPs.  Each  student  accumulates  a minimum  of  ten  hours 
of  classroom  and  hands-on  training  in  air-to-air  combat. 

Two  student  pilots  train  simultaneously  in  the  dual  dome, 
two-cockpit  facility.  Each  student  pilot  is  normally  in- 
structed by  an  individual  IP,  but  a single  IP  can  instruct 
both  pilots  simultaneously.  Training  data  are  normally 
recorded  while  "flying"  against  a target  with  preprogrammed 
flightpaths.  A kill  is  "scored"  by  guns,  heat  missile, 
radar  missile,  or  ground  strike. 


The  student  pilot  undergoes  initial  briefings  and  sim- 
ulator familiarity  sessions  on  the  first  day  of  the  five 
days  of  training.  After  becoming  familiar  with  the  simula- 
tor characteristics  through  the  hands-on  session,  the  stu- 
dent is  "scored"  against  a series  of  canned  target  maneuvers. 
The  student's  initial  performance  is  recorded  by  computer 
.and  stored  on  magnetic  tape. 

The  training  progresses  during  the  week  in  accord  with 
the  TAC  ACES  I training  syllabus.  The  final  day  of  train- 
ing/ the  fifth  day/  consists  of  a second  scoring  session 
with  each  student  pilot  competing  against  canned  target 
maneuvers  as  was  initially  done  on  the  first  day  of  train- 
ing. The  class  training  culminates  by  a double  elimination 
competition,  or  turkey  shoot,  where  each  student  competes 
against  the  others  in  one-on-one  free  engagements  until 
eliminated  or  a winner  is  decided. 

Background  data  are  collected  on  each  pilot  undergoing 
TAC  ACES  I training.  In  addition,  each  student  pilot  is 
asked  to  subjectively  evaluate  the  simulator  performances 
in  comparison  to  the  actual  aircraft.  Subjective  evalua- 
tions of  the  training  effectiveness  and  potential  improve- 
ments are  also  solicited.  These  data  are  recorded  on  ap- 
propriate questionnaire  forms  and  transmitted  to  TAC,  and  a 
copy  remains  on  file  at  Vo ugh t. 

Utilization  of  Data 

The  accumulated  subjective  critiques  of  the  simulator 
performance  and  the  training  evaluations  obtained  from  the 
student  pilots  and  inputs  from  IPs  are  used  both  by  TAC  and 
Vought  in  evaluating  potential  improvements  in  the  simula- 
tor and  simulator  training. 


18 


Si  \ 

Pi 


i 


V. 


o 


< 


<5‘ 

.V 

V'  - 


t 

?' 


1 


The  objective  measures  of  student,  pilot  performance 
are  used  in  obtaining  the  GSI  to  represent  a measure  of 
relative  proficiency  in  air-to-air  combat  in  the  simulator. 

Student  .pilot  background  data  are  used  to  subjectively 
correlate  a pilot's  expected  level  of  proficiency  with  that 
measured  by  the  GSI. 

Experiment  Controls 

The  data  were  collected  for  a sample  of  90  subjects 
during  the  period  of  this  study , under  concisely  defined 
controlled  conditions.  The  study  was  unique  in  the  sense 
that  the  data  had  to  be  collected  within  and  from  the  opera- 
tional training  environment.  The  collection  of  data  under 
these  conditions  also  had  to  be  made  on  a minimum  interface 
and  non-interference  basis  with  the  ongoing  TAG  ACES  I 
training  program.  This  requirement  precluded  the  applica- 
tion of  experimental  controls  in  a classical  sense,  as  found 
in  a laboratory  experiment.  As  a result,  other  methods  of 
control  were  developed  to  function  within  the  restrictions 
imposed  to  provide  some  assurance  as  to  the  fidelity  of  the 
data  collected  and  to  minimize  the  effect  of  undesired 
variables.  This  was  accomplished  by  briefing  each  new  Chief 
IP  (CIP)  as  to  the  mandatory  adherence  by  IPs  and  students 
to  the  approved  TAC  ACES  I Training  Syllabus.  A form  was 
developed  and  completed  after  each  training  class,  certify- 
ing ho  the  adherence  to  the  TAG,  ACES  I -Training  Syllabus, 
fidelity  of  the  air  combat  simulator  performance,  and  per- 
formance accuracy  of  the  software  and  computer  hardware. 

Data  collected  from  TAC  ACES  I students  prior  to  this  study 
did  not  have  these  controls. 


19 


s| 

i! 


The  TAC  ACES  I students  in  the  study  were  not  aware  of 
the  GSI  Validation  Study  and  the  purposes  of  data  collection. 
Individual  pilot  performance  data  were  collected  on  Monday 
and  Friday  of  the  training  week  and  during  the  "Turkey 
Shoot"  elimination  contest,  after  completion  of  the  formal 
training  program.  In  addition,  performance  data  were 
collected  for  four  of  the  ,12  classes  on  Wednesday  of  the 
training  week.  The  students  were  also  required  to  complete 
a background  questionnaire  and  an  end-of-course  critique;, 

The  existing  questionnaires  were  modified  to  Obtain  age 
group  and  combat  experience  data.  The  Chief  Instructor 
Pilots  (CIPs)  for  the  TAC  ACES  program  were  required  to  pre- 
dict each  student's  performance  in  the  turkey  .shoot  contest. 
As  each  class  completed  the  formal  traininc;  program,  the  CIP 
was  required  to  rank-order  that  class  of  students  as  to 
their  perceived  standing  at  the  completion  of  the  turkey 
shoot  elimination  contest.  Simulation  or  other  training 
syllabus  anomalies  were-  also  recorded  as  a part  of  the  data 
collection  task  to  aid  in  the  identification  of  outliers  in 
the  data  sets. 

All  of  the  student  pilot  performance  data  were  recorded 
on  magnetic  tape.  All  other  data  from  students'  background, 
course  critiques,  and  CIP  rankings  were  recorded  on  forms 
adapted  to  or  generated  for  the  study.  In  addition,  all  of 
the  student  pilot  performance  data  were  produced  on  hard 
copy  printouts  for  verification  and  preliminary  analyses. 

The  forms  developed  and  used  in  the  study  are  included 
in  Appendix  B.  The  TAC  ACES  I Training  Syllabus  and  the 
turkey  shoot  competition  rules  are  included  in  Appendix  C. 
Mathematical  descriptions  of  the  scoring  computations  for 
each  weapon  simulated  in  the  study  have  been  submitted  to 
the  Flying  Training  Division  of  the  Air  Force  Human  Resour- 
ces Laboratory, 
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II . OBJECTIVES 


Scope 


The  scope  of  this  investigation  is  limited  to  the 
optimization  and  validation  of  the  GSI  system.  The  primary 
product  is  an  assessment  of  the  capabilities  and  limitations 
of  the  GSI  scores  as  indicators  of  pilot  Air  Combat  Maneu- 
vering (ACM)  skill  and  the  determination  of  the  utility  of 
GSI  scores  as  predictors  of  pilot  performance  in  free-engage- 
ment  turkey  shoot  competition. 

Derivation  of  Optimal  Models 

The  empirically  derived  GSI  was  statistically  valida- 
ted to  its  predictive  capability  by  the  use  of  statistical 
analysis  techniques.  An  improved  GSI  predictor  using  the 
four  subjectively  selected  parameters  of  the  empirical  GSI 
was  obtained  by  discriminant  analyses.  A further  improved 
GSI  predictor  was  derived  from  the  expanded  list  of  availa- 
ble candidate  predictor  variables  and  variable  selection 
techniques.  These  improved  predictors  were  validated  with 
data  acquired  from  classes  outside  the  experiment.  Confi- 
dence intervals  on  the  predictors  were  provided.  Further, 
standardized  discriminant  functions  were  provided  to  identi- 
fy the  relative  contribution  of  each  parameter  in  the  de- 
rived predictor  equation (s).  Student  pilot  background  and 
subjective  data  obtained  from  questionnaire  forms  were  input 
with  objective  data  to  obtain  optimal  predictor  models. 


Comparison  With  Expert  Opinion 


Subjective  rankings  of  student  pilots  were  obtained 
from  Instructor  Pilots  and  compared  to  the  derived  GSI 
predictors  and  the  actual  pilot  rankings  obtained  from 
turkey  shoot  results.  These  interrelationships  were  des- 
cribed through  the  use  of  correlation  and  yariance/co- 
variance  matrixes. 

Correlation  With  Previous  Data 


Data  from  classes  undergoing  training  prior  to  this 
experimental  study  were  used  on  a random  selected  basis  to 
obtain  measures  of  GSI  prediction  accuracies.  These  in- 
vestigations are  necessarily  limited  to  the  GSI  as  deter- 
mined from  the  four  subjectively  selected  parameters,  since 
other  objective  data  were  not  on  file. 

Reliability  of  GSI  Scores 

The  reliability  of  the  GSI  was  determined  by  calculat- 
ing confidence  intervals  of  predictions  of  turkey  shoot  rank 
and  corresponding  confidence  levels  of  the  degree  of  cer- 
tainty of  the  predicted  value, 

Edumetric  and  Psychometric  Measurement 

A measure  of  learning  effects  was  obtained  by  statis- 
tically analyzing  data  from  four  classes  specifically  struc- 
tured to  obtain  three  scoring  periods  for  each  student  pilot. 
Measures  of  individual  and  group  learning  were  statistically 
derived  as  a function  of  time  in  training.  These  learning 
rates  were  compared  to  student  pilot  performance  data. 
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III.  ANALYSES 


The  GSI  Score  was  computed  from  data  acquired  during 
the  TAC  ACES  I training  of.  each  class,  normally  on  Monday 
and  Friday.  During  the  GSI  Validation  Study,  a third  set 
of  GSI  data  was  collected  on  Wednesday  for  four  of  the  12 
classes  involved.  GSI  data  are  recorded  nominally  against 
five  canned  targets;  generally,  two  of  the  five  are  cine- 
track  and  the  remaining  three  arc  head-on. 

The  equation  defining  GSI  is, 

GSI  = 4.6  ( 70-MILE RR)  + 0 . 86 (PANG) + (O/D-35) +0 . 5 (180-TTFK)  * 1) 
where ; 

MIL  ERR-  average  mil  error  over  two  cinetrack 
runs  while  R < 3,000  ft. 

PANG  - average  percentage  of  engagement  time  in 
pointing  angle  advantage,  R < 3000  .ft., 
over  two  cinetrack  runs. 

O/D  - average  ratio  of  offensive  to  defensive 

time  against  the  head-on  targets.  Offen- 
sive time  is  the  time  the  target  aircraft 
is  in  the  front  hemisphere  of  the  piloted 
aircraft. 

TTFK  - average  time  to  first  kill  (seconds)  from 
beginning  of  run  until  student  achieves 
first  kill  against  head-on  targets  with  gun 
or  heat  missile. 

The  GSI  Score  itself  is  intended  to  have  a possible 
range  between  zero  and  1,000.  Also,  each  of  the  four  compo- 
nent scores  was  originally  intended  to  contribute  equally  to 
the  index  itself.  Scaling  factors  were  adjusted  from  time 
to  time  as  experience  was  gained  and  when  an  adjustment  was 
considered  appropriate.  The  equation  for  GSI  given  above 
contains  the  scaling  factors  used  over  the  data  collection 
period  of  this  study.  MIL  ERR,  PANG,  O/D,  and  TTFK  are 

23 


MS—** 


referred  Nto  as  the  GSI  component  scores  or  component 
variables  in  this  report. 
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Statistical  Analysis  of  GSI  Data 

The  statistical  analysis  of  the  basic  Monday  and  Fri- 
day GSI  scores  and  the  four  GSI  component  scores  collected 
over  the  12-class  experimental  period  is  presented  in  this 
section. 

Histograms  of  the  GSI  scores  and  the  four  GSI  compo- 
nent variables  (part-scores)  are  provided  in  Figures  2 
through  6,  These  show  the  general  distributional  shapes  of 
each  variable.  The  histograms  for  Monday  and  Friday  for 
each  score  are  provided  on  the  same  page  to  facilitate 
visual  comparison.  In  general,  the  distributions  improve 
from  Monday  to  Friday  (increase  or  decrease  as  appropriate) 
and  the  sample  standard  deviations  become  smaller. 

Scatter  diagrams  for  GSI  and  GSI  component  variables 
for  both  Monday  and  Friday  are  presented  in  Figures  7 
through  11.  The  Y-variable  used  to  construct  these  scatter 
diagrams  is  turkey  shoot  rank.  Turkey  shoot  winners  are 
ranked  one,  runners-up  are  ranked  two,  third  eliminators 
always  receive  a rank  of  5.5,  and  first  eliminators  are 
generally  ranked  7.5.  A visual  examination  of  these  scatter 
diagrams  reveals  no  apparent  trends. 

Early  in  the  analysis,  a second  candidate  Y-variable 
was  considered  to  be  of  possible  interest.  This  was  frac- 
tional wins,  defined  as  the  ratio  of  turkey  shoot  wins  to 
the  total  number  of  engagements  for  a given  student  as  indi- 
cated oh  the  double  elimination  tree  used  to  score  the 
turkey  shoots.  Correlation  coefficients  of  the  four  GSI 
component  variables  to  turkey  shoot  rank  and  fractional  wins 
for  both  Monday  and  Friday  data  are  shown  in  Table  1.  The 
presentation  is  constructed  so  that  the  correlation 
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coefficients  for  Monday  data  are  shown  above  the  main  dia- 
gonal of  each  matrix  and  for  Friday  data  are  below  the  main 
diagonal.  As  can  be  seen,  relatively  strong  correlatipns 
exist  among  the  component  variables  indicating  non-zero  co- 
variances  and  thus  lack  of  independence,  i.e.,  possible 
significant  multicolinearities.  Correlations  between  the 
component  variables  and  turkey  shoot  rank  and  fractional 
wins  are  also  seen  ico  be  very  weak.  Various  regression 
analyses  using  appropriate  variable  selection  techniques 
and  ridge  regression  were  also  conducted  as  part  of  this 
study.  Predictive  capabilities  of  these  regression  models 
were  found  to'  be  very  poor.  This  is  what  might  be  expected 
in  view  of  the  scatter  diagrams  provided. 

In  an  attempt  to  determine  significant  sources  of 
variation' within  the  data,  five  three-way  analyses  of 
variance  were  conducted,  for  GSI  and  the  four  component 
variables.  The  three  sources  of  variation  investigated 
were 

(a)  variation  between  days  (Monday  and  Friday) , 

(b)  variation  .between  turkey  shoot  ranks,  and 

(c)  variation  between  the  classes  which,  contained 
eight  students. 

Table  2 shows  the  results  of  the  analysis  of  the  GSI 
scores.  It  was  found  that  very  significant  differences 
exist  between  Monday  and  Friday  GSI  scores  (The  risk  of 
error  in  saying  a.  significant  difference  exists  when  in  fact 
it  does  not  is  less  than  one  percent) , implying,  of  course, 
that  if  GSI  measures  group  learning,  a significant  increase 
occurs  over  the  five-day  class  period.  This  is  discussed  in 
detail  in  the  section  on  edumetrics.  The  ether  significant 
source  of  variation  (also  significant  at  the  one  percent 
level)  is  between  classes.  It  was  preferred  that  significant 
differences  between  classes  would  not  occur,  as  this 
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TABLE  2 - ANALYSIS  OF  VARIANCE  - GSI  SCORES 


SOURCE  OF 
VARIATION 

DF' 

MEAN  SQ . 

F TEST 

BETWEEN 

DAYS 

997,335 

■ 

997,335 

51.1* ** 

BETWEEN 

RANKS 

58,630 

3 

19,543 

1.00 

BETWEEN 

CLASSES 

655,204 

8 

81,900 

4.20** 

RESIDUAL 

2,557,437 

131 

19,522 

TOTAL 

4,268,606 

143 

■ 

HI 

* significant  at  5%  level 

**  significant  at  1%  level 
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could  tend  to  mask  differences  between  ranks,  as  exhibited 
in  the  data,  if  they  really  existed.  Conversely,  it  was 
desired  that  significant  differences  between  GSI  scores  by 
rank  should  occur.  These  differences  did  not  occur,  and 
this  provides  evidence  as  to  why  the  initial  GSI  score  is  a 
relatively  poor  predictor  of  turkey  shoot  rank.  Figure  7, 
which  shows  scatter  diagrams  of  GSI  scores  versus  turkey 
shoot  rank,  provides  graphic  evidence  as  to  why  significant 
differences  between  GSI  Score  and  rank  do  not  exist,  or  at 
least,  they  cannot  be  detected  from  these  data. 

Tables  3 and  4 present  the  three-way  analysis  of 
variance  tables  for  the  GSI  component  variables.  For  the 
component,  variable  average  mil  error,  significant  differ- 
ences between  ranks  appear  to  exist  at  the  1 percent  confi- 
dence level*  but  no  difference  is  evident  between  days.  A 
difference  is  detectable  between  classes  at  the  5 percent 
level. 

For  the  component  variable  percent  PANG,  significant 
differences  are  evident  at  the  1 percent  level.  There  is  no 
evidence  of  significance  for  variation  between  ranks.  For 
the  component  variable,  offensive  time,  significance  be- 
tween days  are  detected  at  the  1 percent  level.  No  differ- 
ences appear  to  exist  between  ranks  or  ^classes.  For  the 
component  variable  TTFK,  significant  differences  are  de- 
tected at  the  5 percent  level  between  days  and  between 
ranks.  Differences  are  not  evident  between  classes.  Table 
5 summarizes  the  finding  of  the  analyses  of  variance  per- 
formed of  the  four  GSI  component  variables. 
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TABLE  3 - GSI  COMPONENT  ANALYSIS  OF  VARIANCE 


ANALYSIS  OF  VARIANCE  - AVERAGE  MIL,  ERROR 


SOURCE  OF 
VARIATION 

• SUM  OF  SQ 

DF 

MEAN  SQ. 

F TEST 

BETWEEN 

DAYS 

152.11 

1 

152.11 

.51 

BETWEEN 
' RANKS 

4,567.06 

3 

1,522.35 

5.15** 

BETWEEN 

CLASSES 

5,568.72 

8 

696.09 

2.35* 

RESIDUAL 

38,764.33 

131 

295.91 

TOTAL 

49,052.22 

143 

ANALYSIS  OF  VARIANCE  - % PANG 


SOURCE  OF 
VARIATION 

SUM  OF  SQ. 

DF 

MEAN  SQ. 

F TEST 

BETWEEN 

DAYS 

2,871.17 

1 

2,871.17 

24.5** 

BETWEEN 

RANKS 

356.08 

3 

118.69 

1.01 

BETWEEN 

CLASSES 

4,114.25 

8 

514.28 

4.38** 

RESIDUAL 

15,371.44 

131 

117.34 

TOTAL 

22,712.94 

143 

* significant  at  5%  level 

**  significant  at  1%  level 
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TABLE  4 - GSI  COMPONENT  ANALYSIS  OF  VARIANCE 


ANALYSIS  OF  VARIANCE  - % OFFENSIVE  TIME 


SOURCE  OF 
VARIATION 

SUM  OF  SQ. 

DF 

MEAN  SQ. 

F TEST 

BETWEEN- 

DAYS 

6,696.69 

1 

6,696.69 

47.2* ** 

BETWEEN 

RANKS 

274.25 

3 

91.42 

- .64 

BETWEEN 

CLASSED 

1,332.47 

8 

166.56 

1.17 

RESIDUAL 

18,600.56 

131 

141.99 

TOTAL 

26,903.97 

143 

ANALYSIS  OF  VARIANCE  - TIME  TO  FIRST  KILL 


SOURCE  OF 
VARIATION^ 

SUM  OF  SQ. 

DF 

MEAN  SQ. 

BETWEEN 

DAYS 

19,113.07 

1 

19,113.07 

BETWEEN 

RANKS 

13,215.75 

3 

4,405.25 

BETWEEN 

CLASSES 

10,873.01 

8 

1,359.13 

RESIDUAL 

107,942.50 

131 

1 

823.99 

TOTAL 

151,144.33 

143 

* significant  at  5%  .level 

**  significant  at  1%  level 
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TABLE.  5 - SUMMARY  OF  RESULTS  OF  ANALYSIS  OF 

VARIANCE  OF  GSI  COMPONENT.  VARIABLES 


SOURCE  OF 
VARIATION 


MIL  ERR 


% PANG 


% O/D 


TTFK 


■BETWEEN  DAYS 
BETWEEN  RANKS 
BETWEEN  CLASSES 


**• 

* 


** 


** 


** 


** 

** 


* significant  at  5%  level 
**  significant  at  1%  level 

A Comparison  of  the  GSI  Predictor 


This  section  presents  a comparison  of  the  best  predic- 
tor using,  the  GSI  Score  as  defined  at  the  beginning  of  the 
study  with  random  selection  and  with  CIP  predictions  (CIPPs) 
made  just  prior  to  the  turkey  shoot  competition.  Compari- 
sons were  made  at  four  levels  of  detail  as  to  the  outcome 
of  the  turkey  shoot  (These  levels  of  detail  are  carried 
throughout  the  remainder  of  the  study) . The  four  levels  are 
defined  as  follows: 

1.  Four  Groups  - Proper  placement  into  the  proper 

turkey  shoot  quartile,  i.e.,  1 or 
2 in  the  first  group,  3-4  in  the 
second  group,  5-6  in  the  third  group 
and  7-8  in  the  fourth  group, 

2.  Upper  Half  Proper  placement  of  students  in  the 

top  four  turkey  shoot  ranks  in  those 
ranks,  i.e,,  1,  2,  3’,  3.5  or  4 in 
these  ranks. 

3.  Winner  and  Proper  placement  of  the  winner  or 

runner-up  in  the  winner/runner-up 
group . 


Upper  Naif 
of  Class  - 


Winner  and 
Runner-Up  - 
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4 . Winner  - 


Proper  identification  of  the 
actual  turkey  shoot  winner. 

The  results  of  this  comparison  are  provided  in  Table  6. 
Note  that  CIPPs  were  not.  made  for  the  first  few  classes 
of  the  experiment;  thus,  only  67  out  of  a possible  90  CIPPs 
were  made.  The  random  selection  probabilities  were  deter- 
mined under  the  assumption  of  independent  random  assignment 
of  students  to  turkey  shoot  position.  For  example,  there 
are  eight  possible  assignments  of  outcome  to  the  turkey 
shoot  position.  One  of  these  positions  is  the  winner  posi- 
tion; another  is  the  runner-up  position;  two  are  third 
eliminator  positions,  etc.  Thus,  the  probability  that  a 
given  student  will  be  assigned  the  winner  position,  given 
that  his  assignment  is  at  random  and  independent  of  all 
other  assignments,  is  one  out  of  eight  or  12.5  percent. 
Similarly,  if  the  grouping  being  considered  is  winner  and/or 
runner-up,  there  are  two  out  of  eight  possible  assignments 
in  this  group.  Therefore,  under  the  same  assumption,  the 
probability  that  a given  student  will  be  assigned  to  the 
winner  and  runner-up  grouping  is  two  out  of  eight  or  25 
percent.  Similar  logic  is  used  in  determining  the  probabi- 
lities associated  with  the  random  assignments  to  the  other 
two  groups. 

Four  entries  are  provided  for  CIPP  and  GSI  ranking 
predictors  for  each  of  the  four  groupings.  These  provide 
basic  data  on  the  actual  predictions.  For  example,  for 
CIPP  and  the  "four  groups"  grouping,  the  CIPs  properly 
placed  21  out  of  67  predictions  in  the  correct  groupings 
(1-2,  3-4,  5-6,  or  7-8);  thus,  21  of  67  or  31.3  percent 
were  correctly  classified.  Ninety-five  percent  confidence 
limits  were  calculated  using  these  data  and  were  determined 
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TABLE  6 


A COMPARISON  OF  FRIDAY  GSI  RANK  PREDICTIONS 
WITH  CHIEF  INSTRUCTOR  PILOT  (CIPP)  AND 

RANDOM  SELECTION 


GSI 


RANDOM 

RANKING 

GROUPINGS 

SELECT. 

CIPP 

(FRI. SCORE) 

. NO. CORRECT 
PREDICT. 

- 

21 

26 

FOUR  GROUPS 
(1-2, 3-4, 

5-6, 7-8) 

. TOTAL  NO. 

PREDICT. 

. % CORRECT 
PREDICT. 

25% 

67 

31.3% 

90 

28.9% 

. 95%  CONFI- 
DENCE INT. 

20.2-42.5 

19.5-38.3 

. NO.  CORRECI 
PREDICT. 

- 

24 

2 7- 

UPPER  HALF 

OF  CLASS 
(1,2, 3-4) 

. TOTAL  NO. 

PREDICT. 

. % CORRECT 

PREDICT. 

50% 

34 

70.6% 

46 

58.7% 

. 95%  CONFI- 

DENCE INT. 

55.2-85.9 

44.5-72,9 

. NO.  CORRECT 
PREDICT. 

6 

9 

WINNER  & 
RUNNER-UP 
(1,  2) 

. TOTAL  NO. 

PREDICT. 

. % CORRECT 

25% 

17 

35.3% 

23 

39.1% 

PREDICT. 

. 95%  CONFI- 

DENCE INT. 

- 

12.6-58.0 

19.2-59.1 

. NO.  CORRECT 
PREDICT. 

- 

1 

3 

. TOTAL  NO. 

- 

9 

12 

WINNER 

(1) 

PREDICT. 

. % CORRECT 

12.5% 

11.1% 

25.0% 

PREDICT. 

, 95%  CONFI- 

— . 

0-31.6 

0-49.5 

DENCE  INT. 
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to  be  20.2  percent  and  42.5  percent'*-.  Thus,  over  the  long 
run,  95  percent  of  the  CIPPs  can.  be  expected  to  be  between. 
20.2  and  42.5  percent  correct.  Similar  information  is  pro- 
vided for  the  other  CIPP  and  the  GSI  ranking  predictors. 
Each  CIPP  and  GSI  ranking  prediction  was  subjected 

to  a test  of  the  hypothesis  that  it  is  equal  to  or  better 

2 

than  random  selection  . The  CIPP  for  the  upper  half  of  the 
turkey  shoot  was  found  to  be  significantly  better  than  ran- 
dom selection  at  the  5 percent  confidence  level.  The  GSI. 
ranking  predictor  was  found  to  be  significantly  better  than 
random  selection  for  winner  and  runner-up  also  at  the  5 
percent  confidence  level.  All  other  predictions  were  found 
not  to  be  significantly  different  from  random  prediction  at 
the  5 percent  level.  Table  7 provides  the  levels  of  signi- 
ficance at  which  differences  would  be  assumed  to  exist. 


TABLE  7 - APPROXIMATE  BISK  LEVEL  * T '..HIGH 

DIFFERENCES  CAN  BE  ASSU'-lED  TO  EXIST 


GROUPINGS 

CIPP 

GSI  RANKING 

FOUR  GROUPS 

15% 

18% 

UPPER  HALF 

5% 

13% 

WINNER  & RUNNER-UP 

26% 

5% 

WINNER 

36% 

20% 

■^Ostle  & Mensing.  Statistics  in  research,  (3rd  ed.)  . Ames: 
Iowa  State  University  Pres"s,  1975,  100-101. 

o 

Ostle  & Mensing.  Statistics  in  research,  (3rd  ed.).  Ames: 
Iowa  State  University  Press,  1975,  l2$-133. 
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Thus,  to  this  point  in  the  analysis,  it  can.  be  con- 
cluded that  CIPPs  can  classify  students  as  to  whether  or 
not  they  will  finish  in  the  upper  half  of  the  turkey  shoot 
with  about  55  to  86  percent  accuracy  while  a simple  GSI 
ranking  scheme  can  correctly  predict  turkey  shoot  winner 
and  runner-up  classification  about  39'  percent  of  the  time. 
For  other  predictions  investigated,  the  two  predictors 
appear  to  be  ho  better  than  random  selection.  The  data  in 
Table  6 will  be  carried  forward  for  comparison  with  more 
sophisticated  predictors  developed  from  the  expanded  data 
sets  acquired  from  the  master  data  base  and  through  the  use 
of  discriminant  analysis. 

The  Discriminant  Analysis  - A Discussion  of  the  Analysis 
Performed 


The  GSI  scores,  the  GSI  component  variables,  the  expan- 
ded set  of  candidate  predictor  variables,  and  the  demogra- 
phic data  were  subjected  to  a series  of  discriminant  analy- 
ses using  the  sub-program  DISCRIMINANT  available  as  part  of 
3 • . 

the  SPSS  package  . The  capabilities  of  this  program  were 
useful  in  the  development  of  predictor  equations  from  the 
available  data.  The  purpose  of  this  analysis  was  to  build 
optimal  prediction  models  which  predict  "turkey  shoot"  rank 
from  data  collected  during  the  12  specified  TAC  ACES  I 
classes.  The  models  derived  used  the  Wilks'  Lambda  variable 
selection  criteria  to  select  the  best  candidate  predictor 
variables  from  those  available.  The  models  derived  are 
optimal  within  the  constraints  of  the  analysis  but  are  not 
necessarily  maximal.  A maximal  predictor  model  could  only 
be  achieved  if  all  possible  models  were  considered. 

3 

Nie,  Statistical  Package  for  the  Social  Sciences  (SPSS), 
(2nd  ed.) . New  York:  McGraw  Hill,  1975,  434-462. 
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Discriminant  analysis  begins  with  the.  desire  to  sta- 
tistically distinguish  between  two  or  more  defined  groups 
using  information  available  from  sample  -data.  It  was  de- 
sired to  predict  turkey  shoot  winners  using  data  collected 
by  the  simulator  computer  from  each  student  during  the  nor- 
mal course  of  his  training  and  also  from  questionnaires. 

The  groupings  of  interest  were  defined  from  turkey  shoot 
rank.  In  a normal  class  of  eight  student  pilots,  there 
are  always  at  least  five  distinguishable  turkey  shoot 
groupings.  These  are  in  order  from  most  favorable  to 
least  favorable  outcome:  winner  (1),  runner-up  (1),  third 
eliminators  (2) , second  eliminators  (2) , and  first  elimi- 
nators (2)  . 

The  .primary  objective  of  the  analysis  was  to  develop 
predictor  algorithms  for  turkey  shoot  winners;  therefore, 
the  groupings  considered  were  structured  to  investigate  the 
level  of  detail  at  which  winners  could  be  predicted  from 
available  data.  Winners  can  be  defined  in  several  wavs. 

One  winner  class  is  the  absolute  winner  or  undefeated  stu- 
dent in  che  turkey  shoot  . A second  winner  class  is  the 
winner  ap.d  runner-up.  This  grouping  scheme  was  used  with 
somvi  limited  success  in  earlier  Vought  investigations  which 
employed  Friday  GSI  as  the  predictor  variable.  A third 
level  of  detail  is  the  upper  half  of  a class  as  determined 
by  the  turkey  shoot  competition.  In  all,  four  different 
grouping  schemes  were  defined  and  investigated.  These 
are  as  follows : 

1.  Winners  (Group  I)  versus  all  others  (Group  II) 

2.  Winners  and  runners-up  together  (Group  I)  versus 
all  others  (Group  II) 

3.  The  upper  half  of  the  class  (Group  I;  winners, 
runners-up,  and  third  eliminators)  versus  the 
lower  half  of  the  class  (Group  II:  second 
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eliminators  and  first  eliminators). 

4.  Pour  Groupings  (Group  I:  winners  and  runners- 
up.;  Group  IT:  third  eliminators;  Group  III: 
second  eliminators;  Group -IV:  first  elimina — 
tors  ) . 

The  analysis  was  conducted  in  four  parts,  each  part 
being  defined  by  the  candidate  predictor  variable  set  to 
be  used.  The  first  analysis  used  only  Monday  and  Friday 
GSI  scores  as  candidate  predictor  variables.  This  analy- 
sis provided  a measure  of  the  best  prediction  capability 
of  the  GSI  itself.  Both  the  Monday  and  the  Friday  GSI  scores 
were  presented  to  DISCRIM  as  candidate  predictor  variables. 
Thus,  DISCRIM  was  able  to  select  one,  the  other,  or  both 
GSI  scores.  As  it  turned  out  in  the  three  winner  group- 
ings investigated  in  the  first  analysis,  both  GSI  scores 
were  always  included.  The  predictive  capabilities 
determined  hare  were  then  used  as  the  baseline,  or  basis 
of  comparison,  for  the  other  three  analyses  which  followed. 

The  discriminant  analysis  considers  more  tb.an  just 
correct  classification  into  the  desired  group.  Two  groups 
are  defined,  one  group  including  the  winners,  and  the 
other  group  including  the  non-winners.  It  is  possible  to 
correctly  classify  most  of  the  true  winners  but  incorrectly 
classify  some  relatively  large  number  of  non-winners  as 
winners.  It  must  be  decided  how  many  non-winners  can  be 
accepted  in  the  winner  group.  This  study  found  that  by 
using  indicators  more  complex  than  the  GSI  Score  itself, 
it  was  possible  not  only  to  correctly  classify  "winners'1 
a fairly  large  percent  of  the  time,  but  also  to  greatly 
reduce  the  classification  of  non-winners  into  the  winner 
group . 

The  analysis  began  with  the  empirically  determined 
GSI  scores  as  predictor  variables.  In  the  second  analysis, 
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the  four  component  variables,  (or  part  scores)  from  which 
GSI  is  calculated  were  used  instead  of  the  GSI  total  scores. 
The  DISCRIM  program  was  then  allowed  to  select  from  these 
eight  component  variables  (four  for  Monday  and  four  for  Fri- 
day) the  best  predictor  variables  for  each  of  the  four 
classification  schemes.  The  eight  variables-  are  defined  in 
Table  8 which  shows  that  DISCRIM  was  selective  and  never 
used  all  available  data  to  define  the  optimal  prediction 
(classification)  equations. 

Results  of  the  Discriminant  Analysis 

The  results  of  the  four  discriminant  analyses  are  pre- 
sented. Five  pieces  of  information  are  provided  for  each 
discriminant  grouping  scheme: 

1.  A tabulation  of  group  predicted  membership  versus 
o,ctual  group  membership/  using  the  12-class  sample  consi- 
dered in  the  study. 

2.  The  basic  optimal  classification  functions  deter- 
mined by  the  discriminant  program.  These  are  presented  in 
tabular  form.  The  classification  functions  are  used  to  pre- 
dict group  membership.  There  is  one  classification  function 
for  each  defined  discriminant  group.  To  classify  a given 
sample  (case) , the  value  (score)  for  each  classification 
function  is  calculated.  The  sample  (case)  is  then  classi- 
fied into  the  group  for  which  the  classification  function 
provides  the  highest  score. 

30  Standardized  Discriminant  Function (s)  r-  In  this 
study,,  there  is  always  one  less  discriminant  function  than 
the  number  of  groups  defined.  In  general,  the  discriminant 
functions  can  be  thought  of  as  the  axes  of  a geometric 
space,  and  thus  can  be  used  to  study  the  spatial 
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TABLE  f»  " MONDAY  AND  FRIDAY  3S I COMPONENT  VARIABLES  AND 
VARIABLE  SELECTION  BY  DISCRIMINANT  GROUP 


VAR. 

DESIG. 


GROUP  I - Winners-;  GROUP  II  - Others 

GROUP  I - Winners  & Runners-Up;  GROUP  II 
Others  

I GROUP  I - Winners,  R.U.,  & 3rd  Elim.; 
GROUP  II  - Others 


GP.  I - Win.  & R.U. 
GP.  Ill  - 2nd  Elim. 


GP.  II  - 3rd  Elim. , 
GP.  IV  - 1st  Elim. 


VARIABLE  DEFINITION 


X AVERAGE  MIL  ERROR  FOR  FRIDAY 
PERCENT  TIME  IN  PANG  FOR  FRIDAY 
X X PERCENT  OFFENSIVE  TIME  FOR  FRIDAY 
X TIME  TO  FIRST  KILL  ON  FRIDAY  (SECONDS) 


X AVERAGE  MIL  ERROR  FOR  MONDAY 
PERCENT  TIME  Its  PANG  FOR  MONDAY 
PERCENT  OFFENSIVE  TIME  FOR  MONDAY 
TIME  TO  FIRST  KILL  ON  MONDAY  (SECONDS) 
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relationships  among  the  groups.  The  standardized  dis- 
criminant functions  perform  the  same  general  functions  as 
the  -standardized  (beta)  coefficients  in  regression  analy- 
sis. These  functions  provide  ah  easy  reference  as  to  the 
relative  contribution  of  each  of  the  selected  discriminant 
predictor  variables. 

4.  Unstandardized  Discriminant  Functions  — The  un- 
standardized discriminant  functions,  like  the  standardized, 
are  useful  in  the  descriptive  analysis  of  spatial  relation- 
ships among  the  groups. 

5.  Canonical  Correlation  Coefficients  of  the  Discri- 
minant Function (s)  — The  canonical  correlation  coefficient 
provides  an  indication  of  the  relative  capability  of  the 
associated  discriminant  function  to  separate  data  into 
correct  groups.  A value  of  one  indicates  perfect  group 
separation  capability;  a value  of  zero  indicates  total  in- 
ability to  separate  groups. 

The  First  Discriminant  Analysis  - Assessment  of  the  GSI 
Scores  as  Turkey  Shoot  Placement  Predictors 

The  results  of  the  first  discriminant  analysis  are 
presented  in  Tables  9,  10,  and  11,  where  Monday  and  Friday 
GSI  scores  are  the  predictor  variables.  While,  in  general, 
members  of  the  first  group  are  correctly  classified  on  the 
order  of  6C  percent  of  the  time,  many  non-first  group  stu- 
dents are  classified  incorrectly  in  the  first  group.  The 
lack  of  discriminant  power  is  evidenced  by  the  low  values 
of  the  canonical  correlation  coefficients  of  the  respective 
discriminant  functions,  i.e. , between  0.120  and  0.218. 
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TABLE  9 - GSI  TURKEY  SHOOT  WINNER  PREDICTIONS 


PREDICTOR' VARIABLES:  Monday  and  Friday  GSI  Scores 


ACTUAL  GROUP 
MEMBERSHIP 

NO.  OF 
CASES 

Turkey  Shoot 

Winners  ***** 

12 

Turkey  Shoot  Non- 
Winners  (Others)  GFII 

78 

PREDICTED  GROUP  MEMBERSHIP' 

GROUP  I 

GROUP  II 

8 

66,7% 

4 

33.3% 

■ 

34 

3.6% 

44 

56.4% 

VARIABLE 


RRECTLY  GROUPED 


CLASSIFICATION  FUNCTION 
COEFFICIENTS 


0.03594 


0.00581 


0.75773 


CONSTANT  1-13.77014 


0.00757 

0.00437 


2.80178 


CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.140 
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TABLE  10  - GSI  TURKEY  SHOOT  WINNER  AND 
RUNNER-UP  PREDICTIONS 


PREDICTOR  VARIABLES:  Mon 

day  and  Friday  GSI  Scores 

ACTUAL  GROUP 

-NO.  OF 

PREDICTED  GROUP  MEMBERSHIP' 

MEMBERSHIP 

CASES 

GROUP  I 

GROUP  II 

Turkey  Shoot 

Winners  & Runners  GPI 
-Up, 

23 

14 

60.9% 

- "g 

39.1% 

Third,  Second,  and 
First  Elimina-  GPII 

tors  (Others) 

67 

33 

49.3% 

34 

50.7% 

% OF  CASES  MERE  CORRECTLY  GROUPED 
f CLASSIFICATION  FUNCTION 


VARIABLE  1 COEFFICIENTS  I COEFFICIERTS 


GROUP  I 

\ GROUP  n 

STANDARDIZED  . 

[^STANDARDIZED 

FGSI 

0.03804 

0.0396 

0.96359 

0.00759 

0.00055 

MGS  I 

0.00596 

0.00581 

0.09485 

CONSTANT 

-13.85377 

-12.45815 

— 

-5.02757 

■ — 



| 

J 

I 

CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0 . 120 
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TABLE  11  - GSI  TURKEY  SHOOT  WINNERS-,  RUNNERS-UP  AND  THIRD 
ELIMINATOR  PREDICTIONS  (CLASS  UPPER  HALF) 


PREDICTOR  VARIABLES:  Monday  and  Friday  GSI  Scores 


ACTOAL  GROUP 

NO.  OF 

PREDICTED  GROUP  MEMBERSHIP 

MEMBERSHIP 

CASES 

GROUP  2 

GROUP  II 

T.S.  winners, 
Runners-Up  and  GPI 

46 

27 

19 

Third  Eliminators 

58.7% 

41.3% 

T.S.  Second  and 

19 

25 

First  EiiminatorsGPII 

44 

43.2% 

56.8% 

57.6  % OF  CASES  WERE  CORRECTLY  GROUPED 

VARIABLE 

FGSI 

MGS  I 


CLASSIFICATION  FUNCTION 
COEFFICIENTS 


0.03764 

0.00759 


CONSTANT  1-14.06189 


0.03587 

0.00574 

-12.11746 


DISCRIMINANT  FUNCTION 
COEFFICIENTS 


STANDARDIZED 

0.49739 

0.71398 


^STANDARDIZED 

0.00392 

0.00412 

-4.32199 


CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0-218 


The  Second  Discriminant  Analysis  - Statistical  Deriva- 
tion df  an  Optimal  Four  Parameter  Predictor  - Derives 
Optimal  Predictors  Using  the  Same  Four  Parameters  of  the 
Empirically  Derived  GSI  Scores 

The  results  of  the  second  discriminant  analysis  are 
presented  in  Tables  12,  13,  14!,  and  15.  In  this  analysis, 
the  eight  GSI  component  variables  (four  for  Monday  GST 
component  scores  and  four  for  Friday  GSI  component  scores) 
are  used  as  candidate  predictor  variables  (Table  8) . The 
table  for  each  group  definition  indicates  the  variables 
selected  by  DISCRIM.  For  example,  X3  and  X8  (Percent 
Offensive  Time  for  Friday  and  Time  to  First  Kill  (TTFK)  for 
Monday,  respectively)  were  selected  by  DISCRIM  for  inclusion 
in  the  analysis  where  the  12  turkey  shoot  winners  comprise 
the  top  discriminant  group.  The  predictive  capabilities  of 
this  analysis  appear  to  be  marginally  better  than  in  the 
GSI  score  analysis.  The  second  analysis  also  investigated 
four  groupings  (quartile  ranking) (Table  15) . The  standard- 
ized and  unstandardized  discriminant  function  coefficients 
are  also  presented  in  Table  15. 

The  Third  Discriminant  Analysis  - Statistical  Deriviation 
of  Turkey  Shoot  Placement  Predictor  frcm  an  Expanded 
objective  Data  Set 

The  results  of  the  third  discriminant  analysis  are  pre- 
sented in  Tables  16,  17,  18,  and  19.  Candidate  predictor 
variables  were  developed  from  the  complete  objective  data 
set  collected  during  the  Monday  and  Friday  GSI  scoring 
session  but  previously  not  analyzed.  The  table  for  each 
group  definition  indicates  the  predictor  variables  selected 
for  the  given  grouping  scheme.  The  expanded  set  of  candi- 
date variables  and  their  definitions  are  contained  in  Table 
20,  The  canonical  correlations  of  the  discriminant 
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TABLE  12  - OPTIMAL  FOUR  PARAMETER  TURKEY  SHOOT  WINNER 
PREDICTORS 

PREDICTOR  VARIABLES:  Monday  and  Friday  GSI  Component 


ACTUAL  GROUP 
MEMBERSHIP 


TurKey  Shoot 

Winners  12  7 

Turkey  Shoot  Non^ 

Winners  (Others)  7b  4 

61.1  % OF  CASES  WERE  CORRECTLY  GROUPED 

CLASSIFICATION  FUNCTION 
VARIABLE  COEFFICIENTS 


Variables 

- 

NO.  OF 

PREDICTED  GROUP  MEMBERSHIP 

CASES 

GROUP  I 

GROUP  II 

GPI 

12 

9 

75.  OS 

3 

25.0% 

am 

7b 

32 

41.03 

46 

59.0% 

GROUP  I 

>:3_ 1.25471 

Xb 0.14417 

Constant  -55.04840 


DISCRIMINANT  FUNCTION 
COEFFICIENTS 


0.09978 

0.01968 

-9.32037 


■ ■ ,.Y.  f.  .t.x.*  --  K t. - 

1.18593 

0.13060 

0.78341 

0.69032 

-48.45673 

— 

- 

CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.226 


TABLE  -13  - OPTIMAL  FOUR  PARAMETER  TURKEY  SHOOT  WINNER 
AND  RUNNER-UP  PREDICTOR 


PREDICTOR'  VARIABLES : Monday  and  Friday  GSI  Component 

Variables 


ACTUAL  GROUP 

NO.  OF 

PREDICTED  GROUP  MEMBERSHIP 

MEMBERSHIP 

CASES 

GROUP  II 

Turkey  Shoot 

Winners  & Runners 6?S 
_iln 

23 

15 

65.2% 

8 

34.8% 

Third,  Second  & 

First  Eliminators6?^ 


29 

43.3% 


% OF  CASES  WERE  CORRECTLY  GROUPED 


VARIABLE 


0.17609 


0.20701 


0.19877 

0.21484 

0.00750 

0.01261 

-19.45735 

-23.34183 

DISCRIMINANT  FUNCTION 
COEFFICIENTS 

1 

-0,74791 

-0.04149 

-0.45291 

-0.04032 

-0.51216 

-0.02157 

-0.42452 

-0.00686 

5.38022 


CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.301 


TABLE  14  - OPTIMAL  FOUR  PARAMETER  TURKEY  SHOOT  WINNER, 
RUNNER-UP  & 3RD  ELIMINATOR  PRED.'( UPPER  HALF) 


raEDICTOR  VARIABLES:  Monday  and  Friday  GSI  Component 

Variables 


HIEDICIED-  GROUP- MEMBERSHIP 


.ACTUAL  GROUP 

*1 

MEMBERSHIP 

T.S. 

Winners , Runners 

Up  & 

Third  Eli-  GPI 

minators 

GROUP  I 


27 

58.7% 


GROUP  II 


28 

63.6% 


VARIABLE 


OF  CASES  WERE  .CORRECTLY  GROUPED 


CLASSIFICATION  FUNCTION 
COEFFICIENTS 


2.23993  I 2.20200  | 0.37015  | 0.04714 

-0.01671 


0.04012  0.05417  I -0.61199  -0.01745 


0.64692 


CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.357 


TABLE  15  - OPTIMAL  FOUR  PARAMETER'  QUARTILE  RANK  PRE- 
DICTORS 


PREDICTOR  VARIABLES:  Monday  and  Friday  GSI  Component 

Variables 


ACTUAL  GROUP 
MEMBERSHIP 


Turkey  Shoot  Gp  1 

Winners  & Runners  Up 

Turkey  Shoot  Gp 

Third  Eliminators 


NU,  predicted  GROUP  MEMBERSHIP 

OF  — : — ^ 

ASEi  GP  I GP  ,ir  -GP  ~III  GP  IV 
10  5 4 4 

23  43.5%  21.7%  17.4%  17.4 

” T 13  ~~1  5 


Turkey  Shoot 
Second  Eliminators 

Turkey  Shoot 
First  Eliminators 


GP  III 


GP  IV 


13 

56.5% 

1 

4.3% 

5 

21.7% 

4 

5 

7 

17.4% 

-21.7% 

; 30.4% 

3 

6 

8 

14.3% 

1 28.6% 

38,1% 

40.0%  OF  CASES  WERE  CORRECTLY  GROUPED- 

I CLASSIFICATION  FUNCTION  COEFFICIENTS 


VARIABLE 


GROUP  I 
0.14229 
1.26789 
0.01537 
0.10084 


GROUP  II  GROUP  III 


0.18733 

1.27884 

0.02250 

0.08284 


0.16523 

1.23831 

0.01595 

0.10708 


GROUP  IV 
0.14452 
1.17700 
0.02180” 
0.10920 


Constantfi  -53.39603  I -53.99324  | -52.95479 


TABLE  16  - EXPANDED  OBJECTIVE  PARAMETERS  TURKEY  SHOOT 
WINNER  PREDICTORS 


PREDICTOR  VARIABLES:  Expanded  Data  Set  (Without  Demographic 

Data) 

ACTUAL  GROUP 
MEMBERSHIP 

NO.  OF 
CASES 

PREDICTED  GROUP  MEMBERSHIP 

GROUP  I , 

GROUP  II 

Turkey  Shoot 

Winners  GPI 

12  . 

10 

83.3% 

2 

16.7% 

Turkey  Shoot 
Non-Winners  GPII 

(Others) 

. 11 

9 

11.7% 

68 

88.3% 

87.6  % OF  CASES  WERE  CORRECTLY  GROUPED 

VARIABLE 

CLASSIFICAT3 

COEFFIC 

. .GROUP  I 

ON  FUNCTION 
IENTS 

group  rr 

DISCRIMINAN 

COEFFIC 

STANDARDIZED 

T FUNCTION 

IENTS 

UNSTANDARDIZED 

M8 

0.00575 

. 0.00755 

0.23698 

0.32092 

-0.00063 

0.01511 

M12 

0.86869 

0.82532 

M16 

1.58034 

1.19862 

0.15558 

0.13295 

M2  9 

0.18453 

0.23185 

-0.21587 

-0.01648 

M32 

0.02928 

•0.02361 

0.82497 

0.00197 

Fll 

1.39074 

0.61329 

0.80084 

0.27081 

F18 

0.05870  • 

0.16967 

-0.38896 

-0.03865 

F22 

-0.10910 

0.72217 

-0.74215 

-0.28957 

F23 

0.15750 

0.09483 

0.45025 

0.02183 

F27 

4.35721 

4.77215 

-0.20194 

-0.14455 

F?9 

0.35718 

0.31550 

0.19126 

0.01452 

CONSTANT 

-118.97914 

-116.51265 

— 

-0.20297 

. 



- 



CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.617 
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TABLE  17  - EXPANDED  OBJECTIVE  PARAMETERS  TURKEY  SHOOT 
WINNER  AND  RUNNER-UP  PREDICTORS 


PREDICTOR  VARIABLES:  Expanded  Data  Set  (Without  Demographic 

Data) 


ACTUAL  GROUP 
MEMBERSHIP 


NO.  OF 
CASES 


Turkey  Shoo* 
Winners  and 
nners-U 


Thi 
and 
t 


PREDICTED  GROUP  MEMBERSHIP 


GROUP  II 


19 

82.6% 


16 

24.2% 


50 

78.8% 


77.5  % OF  CASES  WERE  CORRECTLY  GROUPED 


VARIABLE 

“ 

CLASSIFICATION  FUNCTION 
COEFFICIENTS 

DISCRIMINANT  FUNCTION 

COEFFICIENTS 

1 

STANDARDIZED 

UNSTANDARDIZED 

M10 

0.02022 

0.02543 

0.18801 

-0.0030.2 

Ml  4 

0.00224 

0.00245 

-0.19653 

-0.00012 

M2  4 

0.43126 

-0.28432 

-0.02427 

M2  9 

0.13310 

0.17617 

-0.32723 

-0.02498 

M3  2 

-0.00597 

-0.00758 

0.39114 

0.00094 

F18 

0.42068 

0.51226 

-0.53455 

-0.05311 

F27 

5.90280 

6.40625 

-0.40832 

-0.29227 

F29 

0.12566 

0.05530 

0.53757 

0.04080 

CONSTANT 

-54.73753 

— 

6.15123 

CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0 .542 
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TABLE  18  - EXPANDED  OBJECTIVE  PARAMETERS  TURKEY  SHOOT 
WINNERS,  RUNNERS-UP  AND  THIRD  ELIMINATOR  r(UPPER  HALF)  PRED. 


PREDICTOR  VARIABLES:  Expanded  Data  Set, (Without  Demographic 

Data 


PREDICTED  GROUP  MEMBERSHIP 


ACTUAL  GROUP 
MEMBERSHIP 

NO.  OF 
, CASES 

T.S.  Winners, 
Runners-Up  and  GPI 

Third  Eliminators 

46 

T.S.  Second  and 

First  Elimina-  GPII 

tors 

43 

GROUP  I 


10 

23.3% 


GROUP  II 


10 

21.7% 


33 

76.7% 


77.5  % OF  CASES  WERE  CORRECTLY  GROUPED 


VARIABLE 


CLASSIFICATION  FUNCTION 
COEFFICIENTS 

ft 

0 .02266 

0.02134 

0.51193 

0.59123 

0.08904 

0.05773 

20.40007 

21.28377 

0.21914 

■ebbeeii 

0.12007 

0.09632 

0.01772 

-0.04312 

-0.56101 

■iHMiMII 

m i 

DISCRIMINANT  FUNCTION 

COEFFICIENTS 

STANDARDIZED 

-0.22960 

-0.00068 

*0.27909 

0.04063 

-0.37943 

-0.01603 

0.30075 

0.45352 

0.47031 

0.04673 

-0.25523 

-0.01216 

-0.41056 

-0.03116 

0.30273 

0.59262 

CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.616 
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TABLE  19  - EXPANDED  OBJECTIVE  PARAMETERS  QUARTILE 
RANK  PREDICTORS 


PREDICTOR  VARIABLES:  Expanded  Data,  Set  (Without 

Demographic  Data) 


ACTUAL  GROUP 
MEMBERSHIP 


Turkey  Shoot  1 

Winders  & Runners  Up 


PREDICTED  GROUP  MEMBERSHIP 


GP  IV 


Turkey  Shoot 
Third  Eliminators 

Turkey  Shoot 
Second  Eliminators 

/• ,v .. 

Turkey  Shoot 
First  Eliminators 


GP  II 


GP  III 


GP  IV 


14 

60.9% 

4: 

17.4% 

5 

21.7% 

1 

5.6% 


4 

‘17.4% 

.13 
'5  6.  ,5% 

3 

. 13.0% 

0 

0.0% 


3 

13.0% 

12 

52.2% 

2 

10.0% 


3 

13.0% 

3 

13.0% 

3 

13.0% 

17 

85.0% 


62.9  % OF  CASES  WERE  CORRECTLY  GROUPED 


CLASSIFICATION  FUNCTION  COEFFICIENTS 


VARIABLE 

GROUP  I 

GROUP  II 

M9 

0.39080 

0.43457 

Mil 

-1.09244 

-1.19695 

M2  2 

3.76039 

3.83710 

M2  5 . . 

0.05883 

0.07826 

M32 

-0.00712 

-Q. 00953 

FI 

23.15227 

22,54955 

FI6 

1,25992 

1,56965 

F/18 

6 : 38089 

0.42928 

/F23 

0.32194 

0 ,,29975 

F25 

0.23929 

0.24372 

F27 

0,43905 

1.20003 

F29 

-0.04349 

-Q. 07499 

CONSTANT 

-134,68774 

-140.15710 

GROUP  III 
0.42527 
-0.84949 
3.54577 
Q. 04751 
~0 .00784 
23.13644 
1,20116 
0.49087 
0,32001 
o':  21273 
0.74263 
-0,06826 
-139,95496 


GROUP  IV 
0.32973 
-1. 27048 
4^03881 

0^05261 

-0.0Q910 

24.77510 

1.61926 

0,49089 

0,28426 
~~~  0,24954 
0.4039-7 
-0 « 20159* 
”14  7 76  2 7 9 3 


TABLE  19  (CONT.)  - EXPANDED  OBJECTIVE  PARAMETERS  QUARTILE 

RANK  FUNCTION  COEFFICIENTS 


PREDICTOR  VARIABLE  SET:  Expanded  Data  Set  (Without '.Demographic  - 

Data) 

VARIABLE 

ST 

FCN . I 

DISCRIMI 

ANDARDIZE 

FCN,  II  ' 

NANT  FUNCG 

D 

FCN.  Ill 

.’ION  COE  FI 

UN£ 

FCN...  I 

riCIENTS 

STANDARD!* 

FCN.  II 

iED 

FCN.  Ill 

A 

M9 

0.29372 

0.061509 

0.50531 

0.02840 

0.00629 

0.04886 

Mil 

0.27379 

0.72985 ' 

0.13650 

0.07279 

0.19405 

0.03629 

M2  2 

-0.28484 

-0.44215 

-0.17302 

-0.11116 

-0.17255 

-0.06752 

M2  5 

0.07615 . 

-0.33074 

0.24529 

0. 00322 : 

-0.01398 

0.01037 

M3  2 

0.21123 

0..27740 

0.52465  . 

0.00051 

0.00066 

-0.00126 

FI 

-0.43789 

0.08832 

-0.47683 

-0  .66033 

0.13319  : 

-0.71904 

F16 

-0.22091 

-0.35071 

0.19407 

r0.. 11332 

-0.17990 

0.09955 

FI  8 

-0.31037 

0.38222 

0.35250 

-0.03084 

0.03799 

mssm 

F23 

0.25134 

0.20717 

-0.12811 

0.01219 

0.01004 

jggjggl 

F25 

-0.-10654 

-0.37477 

-0.13307 

-0.00508 

•».  j ..  . 

-0,.  017  86 

1 

F27 

0.12900 

-0.17522 

0.77547 

0.09234 

-0.12542 

BESB 

F29 

0.72779 

0.10283 

.0.10076 

0.05524 

0 ,00780 

0.00765  | 

CONSTANT 



— 

— 

3.90265 

-0 ; 10029 

CANONICAL 

; 

CORREL. 

0,647 

0.529 

0.440 

i 
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TABLE  20  - CANDIDATE  OBJECTIVE  PREDICTOR  VARIABLES 


DESIGNATION 


DESCRIPTION 


P29 
F12 
F17 
F04 
F06 
F18 
F01 
F25 
M30 
M17 
F09 
M32 
M12 
F27 
F32 
F08 
Ml  3 
F22 
F23 
F02 
Mil 
MO  9 
F31 
MlO 

M2  5 
M16 
Fll 
F30 
F20 
F19 
M2  2 
M20 
M2  9 
F03 
F16 
M2  4 
MO  4 
M14 
MO  8 
M31 


HIT/MISS  H-MISS  SCORE  HON  (H* (H+M) /HON) 
*TIME  TO  FIRST  KILL  (SEC-AVG/HEAD-ON) 
TOTAL  NO.  HITS  HON  (HITS /HON) 

TOTAL  FUEL  USED  (LBS.  AVG/HEAD-ON) 
*PERCENT  OFFENSIVE  TIME  (%  AVG  HD-ON) 
TOTAL  TIME  IN  H-MIS  ENV  CTK  (TIME/CTK) 
MAX  G'S  (MAX/SERIES) 

TOT.  TIME  IN  GUN  ENV.  HON  (TIME/HON) 
HIT/MISS  R-MIS  SCORE  HON  (H* (H+M) /MON) 
TOTAL  NO.  HITS  HON  (HITS/HON) 

TOTAL  TIME  SR  LT  1500  (SEC-AVG/CTK) 
HIT/MISS  GUN  SCORE  (H*TOTAL  RDS/HON) 
*TIME  TO  FIRST  KILL  (SEC-AVG/HEAD-ON) 

G SPREAD  LON  (MAX  G - MIN  G) 

HIT/MISS  GUN  SCORE  (H*TOTAL  RDS/HON) 
TOTAL  ROUNDS  FIRED  (NO.  TOTAL/HEAD-ON) 
*PEKCENT  TIME  IN  PANG  (%  AVG. /CINETRACK) 
TIME  TO  GUN  ENVELOPE  CTK  (TIME/CTK) 

TIME  TO  GUN  ENVELOPE  HON  (TIME/HON) 

NO.  TIMES  OVER  G (TOTAL  SERIES) 

TIME  TO  PANG  (SEC-AVG. /CINETRACK) 

TOTAL  TIME  SR  LT  1500  (SEC-AVG. /CTK) 
HIT/MISS  GUN  SCORE  (H*TOTAL  RDS/CTK) 
*AVG . MILL  ERROR  SR  LT  3000  (MILS-AVG./ 
CINETRACK) 

TOTAL  TIME  IN  GUN  ENV  HON  (TIME/HON) 
T'-i’AL  NO.  HITS  CTK  (HITS/CTK) 

TIME  TO  PANG  (SEC-AVG. /CINETRACK) 
HIT/MISS  R-MISS  SCORE  HON  (H* (H+M) /HON) 
TOT  TIME  IN  R-MIS  ENV  CTK  (TIME/CTK) 

TOT  TIME  IN  H-MIS  ENV  HON  (TIME/HON) 
TIME  TO  GUN  ENV  CTK  (TIME/CTK) 

TOT.  TIME  IN  R-MIS  ENV  CTK  (TIME/CTK) 
HIT/MISS  H-MISS  SCORE  KON  (H* (H+M) /HON) 
TOTAL  FUEL  USED  (LBS.  AVG./CINETK) 

TOTAL  NO.  HITS  CTK  (HITS/CTK) 

TOT  TIME  IN  GUN  ENV  CTK  (TIME/CTK) 

TOTAL  FUEL  USED  (LBS.  AVG/HED-ON) 

DELTA  ENERGY  STATE  CTK  (INIT-END/CTK) 
TOTAL  ROUNDS  FIRED  (NO.  TOTAL/HEAD-ON) 
HIT/MISS  GUN  SCORE  (H  * TOTAL  RDS/CTK) 


* Variables  used  to  compute  GSI  scores. 
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TABLE  21  - EXPANDED  OBJECTIVE  PLUS  DEMOGRAPHIC  PARAMETERS 
TURKEY  SHOOT  WINNER  PREDICTORS 

PREDICTOR  VARIABLES:  Expanded  Set  Including  Demographic 

Data 


ACTUAL  GROUP 
MEMBER  SHU 


HO.  OF 
CASES 


PREDICTED  GROUP  MEMBERSHIP 


Turkey  Shoot 
Winners 


Turkey  Shoot  Non- 
Winners  (Others)  77 


GROUP  I 

GROUP  II 

10 

2 

83.3% 

16.7% 

y 

68 

11.7% 

88,3% 

87.6  % OF  CASES  WERE  CORRECTLY  GROUPED 

CLASSIFICATION  FUNCTION 
VARIABLE  COEFFICIENTS 


.-.GROUP  I 

-0.00135 

- U. 09063 
0.00116 
0.98447 
0.64778 



U.2bJ.24__ 
_ U.  12053 


GROirp  rr 

- O.GOOlu 

'L.iL0  5i^ 

- U 1 1 2 7 3 7 

- 0.00396 

0 , i j 7 . / 


I 't  J J 

:>Uq2 


DISCRIMINAHT  FLECTION 
COEFFICIEHTS 

STANDARDIZED  UNSTANPARDIZED 
0,26212  0,00041 


-iL265(t;> 

0..'1 


._3.55J.3_ 

0.69227 

-0.49723 


0,00041 

. Ji. 000  70 
U. 02181 
-0.00142 
“03^28614^ 
-0.U8967 

0.03  568 
0.27011 
-0.02411 

-0.01252 
-0, 67260  „ 


CANONICAL  CORRELATION  CF  DISCRIMINANT  FUNCTION  IS  U. 620 
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TABLE  22  - EXPANDED  OBJECTIVE  PLUS  DEMOGRAHIC  PARAMETERS 
TURKEY  SHOOT  WINNER  AND  RUNNER-UP  PREDICTORS 


HffiDICTOR  VARIABLES:  Expanded  Data  Set  Including  Demographic 

Data 


ACTUAL.  GROUP 

NO.  OF 

ElEDICTED  GROUP  MEMBERSHIP 

MEMBERSHIP 

CASES 

GROUP  I 

GROUP  II 

Turkey  Shoot 
Winners  & 

Runners  Uo 

GPI 

23 

19 

82.6% 

4 

17.4% 

Third,  Second  & 
First  Elimina- 
±ors,...(CltherjsJ 

GPU 

66 

12 

18.2% 

54 

81.8% 

82. C % OF  CASES  WERE  CORRECTLY  GROUPED 


0.02341 

0.05347 

0.46854 

0.52321 

0.00171 

0,01239 

0.80229 

0.91792 

0.04429 

0.09930 

- 0.00274 

- 0.00417 

0.64870 

0.11529 

0o34346 

0.43268 

F30 


0.15338 


0.47268 


7.60449 


0.06650 


1,56218 


35355 

74499 


0.42358 


0.22037 


0.25887 


0.30951 


0.28076 


- 0.23232 


0.61456 


0.34989 


0.52312 


0.27220 


- 0.44603 


0.21687 


0.00055 
- 0.00159 


0.01171 


0.02131 


0.00416 


0.04506 


0,02144 


- 0.00056 


0.20782 


0.03477 


0.20411 


0.19484 


- 0.03385 


0.42454 


CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.654 


TABLE  23  - EXPANDED  OBJECTIVE  PLUS  DEMOGRAPHIC  PARAMETERS 
TURKEY  SHOOT  WINNER,  RUNNER-UP  &.  3RD  ELIM.  (UPPER  HALF) 


ACTUAL  GROUP 

NO.  OF 

PREDICTED  GROUP  MEMBERSHIP 

MEMBERSHIP 

CASES 

GROUP  I 

GROUP  II 

T.S.  Winners,  Runners 
Up  & Third  Eli- 

minstnrc 

46 

37 

80.4% 

9 

19.6% 

T.S.  Second  & 

First  Eliminators^^ 

43 

7 

16.3% 

36 

83.7% 

IREDICTOR  VARIABLES;  Expanded  Data-  Set  Including  Demographic 

Data- 


82.0  % OF  CASES  WERE  CORRECTLY  GROUPED 


VARIABLE 

CLASSIFICAT3 

COEFFIC 

GROUP  I 

ON  FUNCTION 
IENTS 

. .group  n 

DISCRBUNAN 

COEFFIC 

STANDARDIZED 

T FUNCTION 
IENTS 

UNSTANDARDIZED 

. nil 

0.00368 

0.02656 

0.27223 

- 0.19296 

0.01065 

- 0.00057 

....  M4 

0.02228 

0.02106 

M20 

0.52173 

0.62508 

0.33042 

0.04811 

M2  5 

0.00378 

- 0.03865 

- 0.46721 

- 0.01974 

M2  9 

0.33286 

0.36408 

0.19048 

0.01454 

FI 

21.99968 

23.02592 

0.31727 

0.47843 

Fll 

1.48071 

1.35444 

- 0.17362 

- 0.05871 

F18 

0.07518 

0.17284 

0.45738 

0.04545 

F25 

0.21372 

0.18771 

- 0.25382 

- 0.01-209 

F29 

- 0.11230 

- 0.19010 

- 0.47705 

- 0.03621 

F30 

0.21794 

1.52132 

0.30984 

0.60653 

CONSTANT 

-139.34155 

-144.61646 

— 

- 2.44321 

— 

CANONICAL  CORRELATION  OF  DISCRIMINANT  FUNCTION  IS  0.642 
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PREDICTOR  VARIABLES:  Expanded  Data  Set  Including 

Demographic  Data 


ACTUAL  GROUP 
MEMBERSHIP 


Turkey  Shoot  GP  * 

Winners  & Runners  Up 


Turkey  Shoot  GP  11 

Third  Eliminators 


PREDICTED  GROUP  MEMBERSHIP 

GP  I 

GP  II 

-GP  III 

GP  IV 

16 

69.6% 

3 

13.0% 

0 

0.0% 

4 

17.4% 

3 14 

13.0%  60.9% 


% 13.0% 


Turkey  Shoot 
Second  Eliminators 


Turkey  Shoot 
First  Eliminators 


GP  III 


GP  IV 


2 11  4 

8.7%  47.8%  17.4% 


0 2 2 16 
0.0%  10.0%  10.0%  80.0% 


II 


64.0  % OF  CASES  WERE  CORRECTLY  GROUPED 


CLASSIFICATION  FUNCTION  COEFFICIENTS 


VARIABLE 


GROUP  I 


GROUP  II 


GROUP  III 


F29 


CONSTANT 


- 0.00053 

0.00084 

0.00077 

- 0.00063 

- 0.00389 

0.00306 

0 . 46314 

0.48751 

0.48939 

0.00324 

0.01468 

0.00304 

0.58092 

0.61789 

0.93229 

0 . 65554 

0.70357 

0.67359 

1.22072 

1.11891 

0.85281 

0.34258 

0.38840 

0.34136 

1,05453 

1.29887 

1,02676 

0.25768 

0.30279 

0 35183 

0.60374 

0.65370 

0.82041 

0,008308 

0.05169 

0.07938 

0.43946 

0.40639 

0.39330 

-90.43753 

-101.05052 

-97.88803 

GROUP  IV 


0.00314 


- 0.00544 


402 
021 
0.51453 


0,68348 


1.35006 


0.35504 


1.37675 


0.37553 


0.42426 


0.02629 


0.28632 


-94.15775 


TABLE  24  (CONT.) 


| PREDICTOR  VARIABLE  SET;  Expanded  Data  Set  Including 

Denographic  Data 

VARIABLE 

SI 

FCN.  I 

piSCRTM;] 

PANDARDIJII 

FCN.  II 

[NANT  FUNC 

3D 

FCN.  Ill 

HON  COEF 

UN 

FCN.  I 

FICIENTS 

STANDARDI 

.FCN.  II  . 

ZED 

- FCN > III 

D5 

0.67724 

0.06275 

-0,08149 

.0.00106 

0.00010 

-0.00013 

r>6 

-0.60936 

-0.02166 

0.59489 

-0.00130 

-0.00005 

0.00127 

M9 

-0.22317 

0.11192 

-0.38708 

-0.02158 

0.01082 

-0.03743 

M10 

0.33817 

6 . 32989 

-0.16183 

0 .00543 

-6.00530 

-0.00260 

Mil 

-0.16161 

0.78326 

-0.39563 

-0.0497 

0.20825 

-0.10519 

M12 

0 . 1^758 

-0.16675 

-0.58128 

0.00648 

-0.00785 

-0.02737 

M22 

0.16916 

-0.50859 

0.43156 

0 ; 06602 

-0.19848 

0.16842 

M2  5 

0.06428 

-0.42895 

-0.63271 

0.,  00272 

-0.01813 

-0.02674 

F16 

0.19039 

-0.24227 

-0.13338 

0,09766 

-0.12428 

-0.06842 

F18 

0,31104 

0.35562 

-0.14287’ 

0.03091 

0.03533 

-0.01420 

F22 

-0.18372 

0.32385 

-0.31009 

-0.07168 

0.1263G 

-0.12099 

F23 

-0.35169 

_ .0,  25426 

0. 120.47. 

-0.01705 

0.01233 

0.00584 

F29  . _ 

_-0. 59790 

_-Q . 158.39. 

-0.04538.. 

-0 .04538 

-0.00796 

-0.01202 

CONSTANT' 

-0.23871 

-0.59286 

7.72996 

CANON  ICAI 

CORREL. 

0.679 

0.518 

0.450 
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functions  of  the  analyses  have  greatly  increased  over 
analogous  functions  in  the  previous  analysis,  indicating 
increased  capability  to  discriminate  between  groups.  This 
increased  discriminant  capability  is  at  the  cost  of  in- 
creased complexity  in  the  number  of  variables  required  and 
the  complexity  of  calculations.  The  classification  func- 
tions provide  optimal  predictors  for  the  objective  data, 
analyses  in  this  study  and  include  the  best  predictor  varia- 
bles consistent  with  the  Wilks ' Lambda  variable  selection 
criteria.  The  two-group  analyses  (Tables  16,  17,  and  18) 
provide  correct  classification  into  the  top  group  on  the 
order  of  80  percent;  however,  a fairly  large  number  of  non- 
Group  I members  are  still  being  placed  in  these  groups. 

The  Fourth  Discriminant  Analysis  - Statistical  Deriviation 
of  a Turkey  shoot  Placement  Predictor 'Using  Expanded 
Objective  Parameters  Plus  Demographic  Parameters  as  Candi- 
date Variables 


The  results  of  the  fourth  discriminant  analysis  are  pre- 
sented in  Tables  21,  22,  23,  and  24.  The  analysis  uses  as 
candidate  predictor  variables  all  of  the  predictor  variables 
reflected  in  the  third  analysis  plus  seven  candidate  demo- 
graphic variables.  These  specific  demographic  candidate 
variables,  Table  25,  were  available  fcr  all  students;  thus, 
no  sample  size  reduction  was  required, 

TABLE  25  - CANDIDATE  DEMOGRAPHIC  VARIABLES 

DESIGNATION  DESCRIPTION 

114  TOTAL  PILOT  PLIGHT  TIME  (HOURS) 

6 5 TOTAL  PILOT  FIGHTER  TIME 

0&  TOTAL  PILOT  P-4  TIME  (A/C  & IP  HOURS) 

D7  TOTAL  SORTIES  LAST  SIX  MONTHS 

DIO  TOTAL  BFM/ACM  SORTIES 

Dll  BFM/ACM  SORTIES  LAST  SIX  MONTHS 

D13  TIME  SINCE  LAST  BFM/ACM  (WEEKS) 
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The  objective  of  the  fourth  analysis  was  to  investi- 
gate the  possibility  of  reduction  of  mis-classification 
of  cases  into  Group  I while  maintaining  comparable  predic- 
tion rates.  Comparison  of  the  prediction  results  for  the 
fourth  analysis  with  those  of  the  third  indicate  that  the 
fourth  analysis  predictions  were  as  good  or  better  than  the 
third  analysis.  Mis-classification  into  Group  I was  reduced 
in  three  of  the  four  classifications,  and  correct  classi- 
fication into  Group  I was  improved  slightly  in  two  of  the 
fc.ur  classifications.  Evidence  of  this  improved  discrimi- 
nation is  provided  by  improvements  (increases)  in  the 
canonical  correlations  of  the  discriminant  functions. 

In  the  first  classification  scheme  (Group  I - Turkey 
Shoot  Winners,  Group  II  - Other) , the  number  of  predictor 
variables  required  to  maintain  a constant  correct  classifi- 
cation rate  was  reduced  from  11  to  10  by  inclusion  of  demo- 
graphic data. 

Discussion  of  Third  and  Fourth  Analyses 


In  the  third  analysis,  over  80  predictor  variables  were 
available  for  consideration  as  candidates  for  the  analysis. 
These  variables  were  calculated  using  the  master  data  base 
which  Vought  constructed  during  the  first  part  of  this 
study.  These  data  include  the  expanded  list  of  12  variables 
which  were  required  by  the  contract  to  be  analyzed.  An 
initial  screening  of  the  complete  list  was  necessary  to 
reduce  the  number  of  variables  to  acceptable  size. 

This  screening  was  accomplished  by  correlating  all 
variables  with  turkey  shoot  rank  and  then  selecting  the 
40  variables  from  the  list  with  the  greatest  correlation 
coefficients.  The  40  candidate  variables  are  presented  in 


J 
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Table  26  by  rank  as  determined  by  the  absolute  values  of 
the  correlation  coefficient  (R) . Variable  designations  are 
coded  so  as  to  indicate  the  class  day  on  which  each  is 
collected.  For  example,  F29  indicates  that  the  variable 
value  is  collected  on  Friday  (the  "F"  prefix  indicates 
Friday) , whereas  M30  is  a variable  for  which  data  are 
collected  on  Monday.  Table  27  shows  those  objective  varia- 
bles which  were  selected  by  DISCRIM1  as  the  best  turkey  shoot 
rank  predictors.  In  this  table,  the  predictor  variables  are 
separated  by  day  of  data  collection.  The  discriminant 
classification  schemes  by  which  each  are  used  is  also  indi- 
cated. Use  of  this  expanded  list  of  candidate  variables 
appears  to  have  generally  improved  the  winner  prediction 
capability. 

In  the  fourth  analysis,  a selected  set  of  seven  demo- 
graphic variables  were  introduced.  These  were  selected 
mainly  on  the  basis  of  sample  completeness,  as  it  was  not 
desired  to  reduce  the  sample  size  by  excluding  cases  where 
incomplete  data  sets  occurred.  Non-quantitative  data  were 
also  excluded.  All  objective  variables  selected  in  the 
third  analysis  were  retained,  but  objective  data  considered 
in  the  third  analysis  but  not  selected  were  excluded. 

Table  28  defines  the  variables  considered  in  the  fourth 
analysis.  Note  that  "D"  is  the  variable  prefix  used  to  de- 
signate the  demographic  variables  considered.  As  can  be 
seen  from  the  table,  inclusion  of  the  demographic  data  caused 
several  Monday  ("M"  prefix)  variables  to  be  excluded.  Also, 
as  a result  of  the  addition  of  demographic  data  in  the  analy- 
sis, certain  other  variable  selection  changes  occurred. 
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TABLE  26  - CANDIDATE  OBJECTIVE  PREDICTORS  RANKED  BY 

CORRELATION  COEFFICIENT  WITH  ACTUAL  TURKEY 
SHOOT  PLACEMENT 


RANK 

R 

VAR 

DEFINITION 

1 

-.4261 

F29 

HIT/MISS  H-MISS  SCORE  HON  (H* (H+M) /HON) 

2 

+ .316,8 

Fl"2 

*TIME  TO  FIRST  KILL  ( SEC- AVG/HE AD-ON) 

3 

-.3015 

F17 

TOTAL  NO  HITS  HON  (HITS/HON) 

4 

-.2981 

F04 

TOTAL  FUEL  USED  (LBS.  AVG/HEAD-ON) 

5 

-.2957 

F06 

*PERCENT  OFFENSIVE  TIME  (%  AVG  HD-ON) 

6 

r . 2784 

F18 

TOTAL  TIME  IN  H-MIS  ENV  CTK  (TIME/CTK) 

7 

+.2610 

F01 

MAX  G'S  (MAX/SERIES) 

8 

-.2548 

F25 

TOT.  TIME  IN  GUN  ENV  HON  (TIME /HON) 

9 

+.2475 

M30 

HIT/MISS  R-MIS  SCORE  HON  (H* (H+M) /HON) 

10 

-.2382 

M17 

TOTAL  NO.  HITS  HON  (HITS/HON) 

11 

-.2380 

F09 

TOTAL  TIME  SR  LT  1500  (SEC- AVG/ CTK) 

12 

-.2371 

M32 

HIT/MISS  GUN  SCORE  (H*TOTAL  RDS/HON) 

13 

+.2284 

M12 

*TIME  TO  FIRST  KILL  (SEC-AVG/HEAD-ON) 

14 

+.2000 

F27 

G SPREAD  HON  (MAX  G - MIN  G) 

15 

-.1988 

F32 

HIT/MISS  GUN  SCORE  (II*TOTAL  RDS/HON) 

16 

-.1931 

F0  8 

TOTAL  ROUNDS  FIRED  (NO.  TOTAL/HEAD-ON) 

17 

-.1906 

Ml  3 

*PERCENT  TIME  IN  PANG  (%  AVG ./CINETRACK) 

18 

+.1722 

F22 

TIME  TO  GUN  ENVELOPE  CTK  (TIME/CTK) 

19 

-.1677 

F2  3 

TIME  TO  GUN  ENVELOPE  HON  (TIME/HQN) 

20 

+.1666 

F02 

NO.  TIMES  OVER  G (TOTAL  SERIES) 

21 

+.1654 

Mil 

TIME  TO  PANG  ( SEC- AVG , /CINETRACK) 

22 

-.1652 

M0  9 

TOTAL  TIME  SR  LT  1500  (SEC- AVG. /CTK) 

23 

-.1526 

F31 

HIT/MISS  GUN  SCORE  (H*TOTAL  RDS/CTK) 

24 

+.1518 

M10 

*AVG.  MILL  ERROR  SR  LT  3000  (MILS-AVG./ 
CINETRACK) 

25 

-.1435 

M2  5 

TOTAL  TIME  IN  GUN  ENV  HON  (TIME/HON) 

26 

-.1483 

M16 

TOTAL  NO.  HITS  CTK  (HITS/CTK) 

27 

+.1446 

Fll 

TIME  TO  PANG  (SEC-AVG. /CINETRACK) 

28 

+.1437 

F30 

HIT/MI.SS  R-MISS  SCORE  HON  (H*  (H+M) /HON) 

29 

+.1324 

F20 

TOT  TIME  IN  R-MIS  ENV  CTK  (TIME/CTK) 

30 

-.1297 

F19 

TOT  TIME  IN  H-MIS  ENV  HON  (TIME/HON) 

31 

+.1290 

M22 

TIME  TO  GUN  ENV  CTK  (TIME/CTK) 

32 

+.1273 

M20 

TOT.  TIME  IN  R-MIS  ENV  CTK  (TIME/CTK) 

33 

-.1190 

M2  9 

HIT/MISS  H-MISS  SCORE  HON  (H* (H+M) /HON) 

34 

-.1172 

F03 

TOTAL  FUEL  USED  (LBS.  AVG./CINETK) 

35 

-.1111 

F16 

TOTAL  NO.  HITS  CTK  (HITS/CTK) 

36 

-.1108 

M2  4 

TOT  TIME  IN  GUN  ENV  CTK  (TIME/CTK) 

37 

-.0993 

M0  4 

TOTAL  FUEL  USED  (LBS.  AVG/HED-ON) 

38 

+.0908 

M14 

DELTA  ENERGY  STATE  CTK  ( INIT-END/ CTK) 

39 

-.0833 

M0  8 

TOTAL  ROUNDS  FIRED  (NO.  TOTAL/HEAD-ON) 

40 

-.0804 

M31 

HIT/MISS  GUN  SCORE  (H  * TOTAL  RDS/CTK) 

* Variables  used 

to  compute  GSI  scores. 

SELECTED  OBJECTIVE  DISCRIMINANT  VARIABLES 
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VARIABLE  DEFINITIONS 


(N 


H O 

5 D fa 


TOTAL  PILOT  FLIGHT  TIME  (HOURS) 

TOTAL  PILOT  FIGHTER  TIME  (HOURS) 

TOTAL  PILOT  F-4  TIME  (A/C  & IP  HOURS) 
TOTAL  SORTIES  LAST  SIX  MONTHS 
TOTAL  BFM/ACM  SORTIES 
BFM/ACM  SORTIES  LAST  SIX  MONTHS 
TIME  SINCE  LAST  BFM/ACM  (WEEKS) 


TOTAL  FUEL  USED  (LBS.  AVG . /HE AD-ON ) 
TOTAL  ROUNDS  FIRED  (NO.  TOTAL/HEAD-ON) 
TOTAL  TIME  SR. LT. 1500  FT. (SEC. AVG.CTK) 
AVG.  MIL.  ERROR  SR.  LT.  3000  FT.  (MILS- 
AVG./CINETRACK) 

TIME  TO  PANG  (SEC.  AVG . /CINETRACK) 

TIME  TO  FIRST  KILL  ( SEC- AVG/HE AD-ON) 
DELTA  ENERGY  STATE  - CTK  (INT . -END/CTK) 
TOTAL  NO.  -HITS  - CINETRACK  (HITS /CTK) 
TOTAL  TIME  IN  R-MSL  ENV.-CTK  (TIME/CTK) 
TIME  TO  GUN  ENVELOPE-CTK . (TIME/CTK) 
TOTAL  TIME  IN  GUN  ENV.  - CTK  (TIME/CTK) 
TOTAL  TIME  IN  GUN  ENV.  - IIEAD-ON 
(TIME'/H-ON) 

HIT/MISS  HEAT  MIS.  SCORE  - H-ON 
(H* (H+M) /H-ON) 

HIT/MISS  GUN  SCORE  (H*TOTAL  ROS/H-ON) 


MAX  G’S  (MAX/SERIES) 

TIME  TO  PANG  (SEC. -AVG. /CINETRACK) 

TOTAL  NO.  HITS  CINETRACK  (HITS /CTK) 
TOTAL  TIME  IN  H-MIS .ENV. CTK  (TIME/CTK) 
TIME  TO  GUN  ENVELOPE  HON.  (TIME/HON.) 
TIME  TO  GUN  ENVELOPE  HON,  (TIME/HON.) 
TOTAL  TIME  IN  GUN  ENV.  HON.  (TIME/HON.) 
G-SPREAD  HEAD-ON  (MAX.  G-MIN  G OVER 
SERIES) 

HIT/MISS  H-MIS  SCORE  HON  (H* (H+M) /HON) 
HIT/MISS  R-MIS  SCORE  HON  (H* (H+M) /HON) 


Comparison-  of  Prediction  Results 


Table  29  summarizes  the  predictive  capabilities  of  the 
major  predictor  models  presented.  The  table  also  includes 

approximately  95  percent  confidence  limits  on  the  prediction 

4 

rates  . Note  that  the  confidence  limits  are  approximate 
and'  use  the  normal  approximation  to  the  binomial.  This  re- 
quires a relatively  large  sample  size.  For  predictions  of 
the  winner  (the  last  row  of  the  table) , sample  size  is  nine 
or  12. 

Tests  of  the  Predictor  Models 

Given  the  predictor  models  developed  using  discrimi- 
nant analysis,  it  is  necessary  to  test  these  models  using 
data  collected  outside-  the  experimental  data  set.  The 
purpose  of  these  tests  is  to  determine  if  the  predictabi- 
lity of  the  developed  models  is  retained  using  predictor 
variable  data  not  used  in  the  calculation  of  the  parameters 
or  in  the  selection  of  the  predictor  variables.  In  the 
analysis  performed,  there  is  evidence  that  the  parameters 
selected  are  very  sensitive  to  the  particular  data  set  used 
in  their  estimation  and  to  the  definition  of  the  discrimi- 
nant groups.  The  values  of  the  parameter  estimates  are  also 
probably  quite  sensitive  to  the  data  set  used. 

A very  limited  test  analysis  using  data  obtained  prior 
to  this  study  has  been  conducted  on  the  predictor  models 
developed  from  the  first  and  second  analysis  defined  pre- 
viously. In  the  first  analysis,  Monday  and  Friday 
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TABLE  29-  COMPARISON  OF  PREDICTION  RESULTS 


GSI  scores  were  the  predictor  variables.  In  the  second 
analysis,  the  predictor  variables  were  selected  Monday 
and  Friday  GSI  component  variables.  No  additional  analysis 
has  been  conducted  on  outside  data  for  the  third  and  fourth 
analyses  because  it  has  been  determined  that  the  required 
data  are  not  available  and/or  not  available  to  the  extent 
necessary  for  reduction  to  the  master  data  base  form. 

Two  other  difficulties  were  also  encountered  in  ac- 
quiring prior  datn  for  model  testing.  First,  adjustments 
had  been  made  in  the  weighting  factors  used  in  calculating 
GSI.  These  adjustments  were  not  documented,  and  thus,  a 
consistent  set  of  historic  GSI  scores  is  not  readily  avail- 
able. The  second  difficulty  encountered  pertains  to  the 
prior  record  Keeping  procedures  on  GSI  component  variables. 
The  automated  GSI  component  variable  reporting  forms  were 
implemented  beginning  with  TAC  ACES  I Class  #7815.  Thus, 
nominally,  GSI  component  variable  averages  were  not  con- 
sistently recorded  in  a us,?ble  form  prior  to  Class  #7315. 
Further,  Class  #7816  had  missing  data  for  Monday  GSI  com- 
ponent variables.  For  Classes  #7832  and  #7833,  two  classes 
held  after  the  study  sample,  it  was  determined  that  turkey 
shoot  compilations  were  conducted  in  an  irregular  manner; 
that  is,  certain  competitions  were,  terminated  when  two  con- 
testants were  eliminated  simultaneously  by  air-to-air 
collision.  This  practice  preempted  evaluation  oi  turkey 
shoot  results,  usxng  the  method  used  previously  in  defining 
ranks.  Thus,  classification  of  results  could  not  be  deter- 
mined using  definitions  defined  for  the  discriminant  pre- 
dictor model. 

The  results  of  these  data  restrictions  liilt*  the 
analysis  to  four  classes  (7815,  7817,  7818,  ana  7819) , 
totaling  30  students.  It  is  als*.  restricted  to  predictors 
using  GSI  and  GSI  component  variables.  This,  of  course, 
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precludes  evaluation  at  this  time  of  the  best  predictor 
models;  that  is,  those  using  the  expanded  data  set  and 
demographic  data.  Recommendations  are  made  at  the  con- 
clusion of  this  report  that  will  alleviate  these  restric 
tions. 

Evaluation,  of  Predictor  Models  Using  Monday  and  Friday 
GS1  Scores 


The  first  comparison  conducted  was  for  groupings 
where  the  top  group  was  defined  to  be  winners  only  and  the 
second  group  contained  all  others.  Figure  12  graphically 
shows  the  classification  of  the  data  from  the  original 
(experiment)  data.  The  graph  shows  Monday  GSI  (MGSI) 
plotted  versus  Friday  GSI  (FGSIl.  The  line  shown  is  ob- 
tained by  setting  the  Group  I classification  function 
equal  to  the  Class  II  classification  and  solving  for  FGSI 
as  a function  of  MGSI.  All  points  above  the  line  are 
placed  in  Group  I (winners)  while  all  points  falling  below 
the  line  fall  in  Group  II  (others) , Figure  13  shows  a 
similar  plot  of  the  test  data  using  the  same  discriminant 
function  developed  from  the  experimental  data.  A statis- 
tical test  of  the  null  hypothesis  that  the  proportion  of 
correct  classifications  (PE)  using  experiment  data  is  equal 
to  the  proportion  of  correct  classifications  (PT)  using  the 
test  data  was  conducted,  i.e.,  HQ  : PE  = P^  versus  : 

P£  J-  Pj,5  The  null  hypothesis  is  accepted  at  the  95  per- 
cent level. 

Similar  plots  are  presented  (Figures  14,  15,  16,  and 
17)  showing  classifications  of  the  experimental  and  test 


5 
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FGSI 


FGS 


Figure  13,  Classification  diagram  - test  data 

(Group  I = winners;  Group  II  = others) 


data  for  the  other  two  group  definitions  defined  in  the 
first  analysis,  i.e.,  Group  I = winners  and  runners-up 
versus  Group  II  = others,  and  Group  I = winners,  runners- 
up  and  third  eliminators  versus  Group  II  = second  and 
first  eliminators) . Similar  tests  of  hypotheses  were  also 
conducted  and  accepted,  i.e.,  no  difference  in  prediction 
rates  between  the  experimental  and  test  data  were  detected. 

Evaluation  of  Predictor  Models  Using  Monday  and  Friday 
GSI  Component  Variables 

The  second  set  of  comparisons  were  made  using  the  pre- 
dictor models  developed  from  the  second  discriminant  analy- 
sis. The  number  of  predictor  variables  selected  for  the 
models  in  this  analysis  was  usually  greater  than  two.  For 
this  comparison,  tabular  displays  were  selected.  Tables 
30,  31,  32,  and  33  provide  the  results  of  the  test  data 
classifications.  For  example,  Table  30  (GP.  I = Winners, 

GP.  II  = Others) , shows  the  data  (X3  and  X8)  and  the  calcu- 
lated classification  function  scores  xClass  FCN  I and  Class 
FCN  II)  used  to  group  the  cases  (actual  group  membership  is 
also  provided  to  determine  correctness  of  fhe  predictions) . 

As  noted  previously,  a case  is  classified  into  the  group 
with  the  greater  classification  function  score.  For  exam- 
ple, consider  the  first  case  (X3  = 72  and  X8  = 98) , The 
function  I score  is  49.4,  and  the  function  II  score  is  49 *7. 
Since  49.7  is  greater  than  49.4,  the  first  case  is  correctly 
predicted  to  belong  to  Group  II,  i.e.,  others  or  non-winners. 
Of  the  30  predictions  shown  in  the  table,  21  or  70  percent 
were  correct.  This  compares  to  an  estimated  correct  pre- 
diction rate  of  about  61  percent  for  the  experimental  data. 
Testing  the  null  hypothesis  that  the  correct  prediction 
rates  of  the  experiment  and  test  sample  are  equal,  a test 
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TABLE  31  - TEST  OF  GSI  COMPONENT  VARIABLE  PREDICTOR  MODI 
(GP  I = WINNERS  & RUNNERS-UP;  GP  II  = OTHERS) 


UPPER  1/2  VS.  LOWER  1/2  (GP.  I = WINNERS,  RUNNERS-UP 

AND  THIRD  ELIMINATORS;  GP . II  = SECOND  & FIRST  ELIMINATORS) 
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No.  Predictions  = 30  No.  Correct  Predi'c£l6hs 


TABLE  33  - QUARTILE  PLACEMENTS  (GP  I = WINNERS  AMD  RUNNERS-UP;  GROUP 
ELIMINATORS;  GP - III  = SECOND  ELIMINATORS;  GP.  IV  = FIRST 


2 

statistic  of  x = 0.766  was  calculated.  As  this  is  less 
2 

than  x (1/  .95)  = 3.84,  the  hypothesis  cannot  be  rejected. 

Comparisons  of  the  group  prediction  capabilities  of 
the  remaining  three  discriminant  predictor  models  were 
also  compared  to  predictions  made  with  the  test  data  and 
are  provided  in  Tables  31,  32,  and  33.  Tests  of  the  hypo- 
thesis of  equality  of  the  predictions  between  the  experi- 
ment data  and  the  te  ’-t  data  were  also  carried  out.  For 
the  prediction  model  where  Group  I = Upper  Half  and  Group 
II  = Lower  Half,  the  null  hypothesis  was  not  rejected. 
However,  for  the  other  two  predictors,  the  null  hypothesis 
was  rejected  at  the  95  percent  level.  For  the  case  where 
Group  I = Winners  and  Runners-Up,  and  Group  II  = Others, 
the  null  hypothesis  could  not  be  rejected  at  the  99  percent 
level. 

This  leaves  the  four  group  predictor  hypothesis  re- 
jected at  the  commonly  acceptable  levels.  Examination  of 
the  sample  means  and  standard  deviations  of  the  predictor 
variables  used  in  each  data  set  provides  some  evidence 
as  to  why  the  null  hypothesis  was  rejected.  Tables  34  and 
35  show  the  comparisons  of  sample  means  and  standard  devia- 
tions by  predictor  variable,  data  set  (experiment  or  test) 
and  by  discriminant  group.  Inherent  in  the  predictor  model 
requirements  is  that  group  membership  prediction  capability 
requires  that  data  for  which  classifications  are  to  be  made 
should  be  samples  from  the  same  distributions  as  those  used 
to  determine  the  predictor  model  itself.  Comparison  of 
the  means  and  standard  deviations  shows  that  several 
rather  distinct  differences  exist  between  the  experiment 
and  test  data  parameters.  An  example  of  these  distribu- 
tional differences  is  contained  in  Figure  18,  where  X5, 
Average  Mil  Error,  is  compared.  Note  the  great  distri- 
butional differences  between  Groups  II,  III,  and  IV 
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TABLE  34  - COMPARISON  OF  GROUP  MEANS 


DISCRIMINANT 

DATA 

DISCRIMINANT 

VARIABLES 

GROUP 

SET 

XI 

X3 

X5 

X8 

GROUP  I - Winners 

Experiment 

30.4 

70.6 

28.1 

124. 

and  Runners-up 

Test 

42.5 

72.9 

31.1 

103. 

GROUP  II  - Third 

Experiment 

43.2 

70.2 

48.9 

109. 

Eliminators 

Test 

33.4 

70.7 

27.9 

103. 

GROUP  III  - Second 

Experiment 

38.5 

68.4 

34.6 

134. 

Eliminators 

Test 

40.1 

64.8 

44.3 

102. 

GROUP  IV  - First 

Experiment 

33.3 

64.8 

60.6 

137. 

Eliminators 

Test 

26.3 

68.6 

21.9 

109. 

TABLE  35  - COMPARISON 

OF  GROUP  STANDARD 

DEVIATIONS 

DISCRIMINANT 

DATA  1 

DISCRIMINANT 

VARIABLES 

GROUP 

SET 

XI 

X3 

X5 

X8 

GROUP  I - Winners 

Experiment 

10.4 

4.84 

22.3 

30.8 

and  Runners-up 

Test 

8.72 

5.61 

21.8 

30.1 

GROUP  II  - Third 

Experiment 

22.2 

6.75 

45.4 

41.0 

Eliminators 

Test 

22.8 

4.11 

15.9 

21.2 

GROUP  III  - Second 

Experiment 

21.1 

8.29 

25.7 

31.2 

Eliminators 

Test 

45.7 

16.0 

44.2 

37.6 

GROUP  IV  - First 

Experiment 

13.6 

10.0 

113. 

31.5 

Eliminators 

Test 

5.12 

9.00 

8.38 

25.5 
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by  data  set.  While  the  Group  I distributions  match  quite 
well,  the  others  change  shape  radically. 


X.v,  DEMOGRAPHIC  DATA  -ANALYSIS 

The  data  collected  as  a part  of  this  study  were  in  two 
primary  forms:  student  pilot  objective  performance  data  in 
the;  simulator  and  student  demographic  data  coJ.lected  from 
background  surveys,  and  questionnaires,  , This  section  des- 
cribes some  of  the  relationships  that  „’eie  investigated . 
between  the  student  pilot's  demographic/ historical  back- 
ground data  and  his  predicted  or  actual  performance  rn  the 
air  combat  simulator.  The  major  data  source  for  comparison 
was  the  TAC  ACES  I background  survey,  shown  in  Appendix  B, 
which  was  adapted  for  use  in  the  G.SI  study.  The.  questions 
on  this  survey  and  their  responses  were  utilised,  to  form 
the  demographic  data  base.  The  form  was  completed  by  each 
student  in  the  study  sample  (N  = 89) , The  questions  were 
identified  as  demographic  variables  and  tabulated  into  a 
list,  which  is  shown  in  Table  35,  Total  Demographic 
Variables.  This  list  was  reduced  to  consider  for  analysis 
only  those  variables  which  included  a positive,  or  other 
than  zero  response  from  all  of  the  89  subjects  in  the  study. 
These  are  shown  in  Table  36,  and  include  those  factors  which 
were  used  in  both  the  correlation  analysis  and  the  stepwise 
selection  routines. 

Several  methods  were  employed  to  analyze  these  data 
which  were  classified  into  two  groups.  Group  1 consists  of 
that  body  of  data  which  resulted  from  responses  from  all 
89  subjects.  Group  2 consists  of  that  body  of  data  which 
resulted  from  responses  from  differing  numbers  of  subjects 
in  the  sample. 

Group  I Data 

A correlation  analysis  was  employed  to  estimate  the 
functional  relationship  among  the  Group  1 data  or  total 
sample  (N  = 89)  of  subjects  in  the  study. 
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TABLE  35 


TOTAL  DEMOGRAPHIC  VARIABLES 


VARIABLE  RESPONDENTS 

N 

Dl 

STUDENT  PILOT  RANK 

89 

D2 

SQUADRON 

89 

D3 

WING 

89 

D4 

TOTAL  PILOT  FLIGHT  TIME,  HOURS 

89 

D5 

TOTAL  PILOT  FIGHTER  TIME,  HOURS 

89 

D6 

TOTAL  PILOT  F-4  TIME,  A/C  AND  IP,  HOURS 

89 

D7 

TOTAL  SORTIES  LAST  6 MONTHS 

89 

D8 

TYPE  AIRCRAFT  CURRENT 

89 

D9 

PRIMARY  DESIGNATED  OPERATIONAL  CAPABILITY 

89 

DIO 

TOTAL  BFM/ACM  SORTIES 

89 

Dll 

BFM/ACM  SORTIES  LAST  6 MONTHS 

89 

D12 

BFM/ACM  SORTIES  LAST  MONTH 

89 

D13 

TIME  SINCE  LAST  BFM/ACM 

89 

D14 

TYPE  A/A  MISSILES  FIRED 

23 

D15 

FWIC  GRADUATE 

1 

D16 

PREVIOUS  ACES  ATTENDED 

18 

D17 

LAST  AGGRESSOR  DACT  FLIGHT 

89 

D18 

OTHER  VISUAL  A/A  SIMULATORS  FLOWN 

18 

D19 

TOTAL  COMBAT  SORTIES 

19 

D20 

TOTAL  COMBAT  HOURS 

19 

D21 

NUMBER  COMBAT  KILLS 

1 

D22 

NUMBER  HITS  RECORDED 

1 

D23 

NUMBER  SAM  ENCOUNTERS 

4 

D24 

NUMBER  HOSTILE  AIRCRAFT  ENGAGEMENTS 

1 

D2  5 

NUMBER  HITS  RECEIVED 

1 

D27 

OWN  TRAINING  EVALUATION 

89 

D28 

ANY  TRAINING  ANOMALIES 

89 
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TABLE  36  - GROUP  1 DEMOGRAPHIC  VARIABLES 


VARIABLE  RESPONDENTS 

N 

D1 

STUDENT  PILOT  RANK 

89 

D2 

SQUADRON 

89 

D3 

WING 

89 

D4 

TOTAL  PILOT  FLIGHT  TIME,  HOURS 

89 

D5 

TOTAL  PILOT  FIGHTER  TIME,  HOURS 

89 

D6 

TOTAL  PILOT  F-4  TIME,  A/C  AND  IP,  HOURS 

89 

D7 

TOTAL  SORTIES  LAST  6 MONTHS 

89 

D8 

TYPE  AIRCRAFT  CURRENT 

89 

D9 

PRIMARY  DESIGNATED  OPERATIONAL  CAPABILITY 

89 

DIO 

TOTAL  BFM/ACM  SORTIES 

89  i 

Dll 

BFM/ACM  SORTIES  LAST  6 MONTHS 

89  j 

D12 

BFM/ACM  SORTIES  LAST  MONTH 

89  ! 

D13 

TIME  SINCE  LAST  BFM/ACM 

89  I 

D17 

LAST  AGGRESSOR  DACT  FLIGHT 

89 

D2  7 

OWN  TRAINING  EVALUATION 

89 

D28 

ANY  TRAINING  ANOMALIES 

89 

Group  1 data  includes  16  demographic  variables,  each 
with  a sample  size  of  89  data  points.  Each  variable  was 
examined  by  correlation  analysis  techniques  to  determine 
the  extent  of  statistical  relationships,  with  four  simula- 
tor performance  measures  and  one  measure  of  predicted  per- 
formance using  "Expert  Opinion".  The  results  presented  in 
Table  37  indicate  no  statistically  significant  relationships. 
The  table  shows  very  low  correlation  between  each  of  the  16 
demographic  variables  and  with  each  of  the  performance  mea- 
sures shown.  Correlation  coefficients  were  also  computed 
between  the  16  variables  and  each  of  the  four  GSI  part  score 
components  for  both  Monday  and  Friday  data.  Again,  the  re- 
sulting correlation  coefficients  were  equally  as  low. 

Finally,  analysis  was  performed  using  those  classes  and 
subjects  with  Wednesday  data  available.  All  of  the  corre- 
lation matrices  developed  were  submitted  to  the  Flying 
Training  Division  of  the  Air  Force  Human  Resources  Labora- 
tory. Correlation  coefficients  were  computed  using  the 
same  group  of  16  variables  against  each  Wednesday  part- 
score  component  and.  the  total  Wednesday  GGi  score.  The 
Wednesday  data  involved  performance  scores  cf  only  27  sub- 
jects. The  results  again  indicated  very  low  correlation. 

Group  2 Data 

An  item  analysis  was  employed  to  estimate  the  func- 
tional relationships  among  the  responses  to  Group  2 data. 

The  analysis  was  generalized  to  observations  due  to  the 
limits  that  are  imposed  on  statistical  inference  by  very 
small  sample  siies.  Sample  size  in  this  group  ranged  from 
N=1  to  N--22,  Two  of  the  Group  1 variables  were  also  in- 
cluded in  this  analysis:  D-17  Last  Agressor  DACT  flight 
and  D-27  Own  Training  Evaluation. 
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TABLE  37  - CORRELATION  AN/ LYSIS 


TURKEY 

FRACT. 

CHF.IP 

GSI 

GSI 

SHOOT 

WINS 

RANK 

MON 

FRI 

01 

STUDENT 

RANK 

-0.0584 

-0.0272 

-0.1061 

-0.0043 

0.0901 

D2 

SQUADRON 

0.2551 

-0.2454 

0.0136 

0.0109 

0.0117 

D3 

WING 

0.0988 

-0.0881 

0.1664 

-0.1040 

-0.0216 

D4 

TOT. PILOT 
FLIGHT 

TIME , HRS. 

0.1835 

-0.2070 

0, 1202 

-0.1184 

-0.0959 

' 

D5 

TOT.f 1GHTEF 
TIME, HRS. 

0.2597 

-0.3093 

0.0215 

-0.0591 

-0.0254 

D6 

TOT.F- 4 
TIME, HRS. 

0.0436 

-0.1252 

-A , 2400 

0.1051 

0.0074 

D7 

TOT. SORTIES 
LAST  6 MOS. 

0.2684 

-0.2414 

-0 .0361 

-0.0116 

0.0155 

D8 

TYPE  ACFT 

0.3218 

-0.3689 

0.0960 

-0.0692 

0.0433 

D9 

PRIMARY  DOC 

0.3168 

-0.3331 

0.0864 

-0.1100 

0.0271 

DIO 

TOT.BFM/ 
ACM  SORTIES 

0.1352 

-0.1282 

0.1307 

-0.0254 

0.0385 

Dll 

BFM/ACM 

SORTIES 

LAST  6 MOS. 

0.1331 

-0,0859 

-0.0161 

0.0400 

0.1537 

D12 

BFM/ACM 

SORTIES 

LAST  MONTH 

0.0371 

-0.0248 

-0.1099 

0.0800 

0.1878 

D 13 

TIME  SINCE 
LAST  BFM/ 
ACM 

0.0089 

0.0375 

0.0838 

-0.0712 

-0.0357 

D17 

LAST  AG- 
GRESSOR 

DACT  FLT, 

0.0215 

-0.0338 

-0.2251 

0.0773 

-0.0540 

D2  7 

OWN  TRAIN- 
ING EVAL- 
UATION 

0.0595 

-0.0725 

-0.0999 

0.0391 

0.0428 

D28 

ANY  TRAIN- 
ING ANOMO- 
LIES 

-0.1078 

0.0367 

-0.0641 

-0.2249 

-0.2097 
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Variable  .D-17,  identified  as  the  date  of  the  subject's 
last  dissimilar  aircraft  air-to-airr  combat  training  flight,, 
was  included  in  the  investigation  because  of  the  dramatic 
effects  of,  aggressor  training  reported  by  DeLeon  (1977)  . 

Variable  D-27  identifies,  the  student  pilot's  affective 
•evaluation  of  the  perceived  value  of  the  training  he  re- 
ceived. It  was  included  for  additional  analysis  to  help 
identify  outlier  scores  and  to  assess,  the  effect  of  attitu- 
dinal  values  on  performance. 

Quest!  •n/Answer  Rationale 

Type  of  Air-To-Air. Missile  Fired 

''What  A/A  missiles  have  you  fired?"  (D-14) 


AIM  7 


AIM  9 


, AI  M 4 


, NONE 


Twenty-two  of  the  89  subjects  rei\  that  they  had 

experienced  launching  missiles  from  thei>.  .^rcraft.  Vhe 
sample  size  (N  22)  represent-;:  24,7  po*  nit  of  the  popular 
tion.  The  surveys  indicated  Uui  Uic  subjects  had  actually 
fired  the  AIM  4/  AIM  7,  v.  9 or  some  combination  ff  these 
missiles.  The  distrust *<.<*-  of  this  group  is  as  follows; 

N 


AIM  4 

AIM  4,  + 7 

AIM  4 f + 1 

AIM  4,  + 7,  + 9 

AIM  7 

AIM  7,  + 9f 
AIM  9 


nt 


0 

0 

1 

1 

3 

11 

JS 

22 


D DeLeon,? , The  peacetime  evaluation  of  the  pilot  skill 
factor  in  the’ air-to-air  combat."*  Rand  deport  R-2Q7Q-PR. 
January  1977.  r ‘ 
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This  group  of  22  subjects  were  examined.  Sot  their  ;pefr 
for&ance  in-  the  turkey  shoot  elimination.  It  was  found'  that 
three  of  the  22  subjects  were  winners  of  turkey  shoots. 

Also,  seven  subjects  (.30.4  percent)  were  found  to  be  either 
winners  or  first  runners-up,  and  all  seven  had  experience 
firing  both  the  AIM-7  and  AIM- 9-  missile. 

It  was  also  found  that  a total  of  seven  of  the  22  sub- 
jects (30,4  percent)  finished  in  the  last  two  places  in  the 
turkey  shoot.  The  CIP  rankings  were  also  compared  for  this 
group.  Of  the  22  subjects,  two  were  predicted  to  win  the 
turkey  shoot  and  six  were  predicted  to  finish  in  last  place 
by  their  IPs. 

Fighter  Weapon  Instructor  Course  (FWIC) 

"Are  you  an  PtfIC  graduate?  (D-15)  Yes  No  

Of  the  80.  subjects  in  the  study  sample,  only  one  of  the 
students  in  the  TAC  ACES  program  had  completed  Fighter  Wea- 
pon Instructor  Course  (EWIC)  training.  It  was  also  found 
that  there  has  been  a total  of  11  FWIC . graduates  out  of  the 
456  subjects  completing  the  T&C  ACES  trailing. 

The  subject  had  experienced  1700  hours  of  total  flying 
time,  1500  hours  of  fighter  aircraft  time,  and  1300  hours  of 
F~4  flying  time. 

A comparison  of  turkey  shoot  data  shows  that  the  sub- 
ject placed  second  in  the  turkey  shoot  contest.  Both  his 
Honda*  and  Friday  GSI  performance  scores  were  above  700 
points..  Analysis  of  the  Friday  GSI  part  scores,  however, 
did  indicate  a deOine  of  up  to  30  percent  from  the  Monday 
GSI  part  scores. 


Previous  ACES  Attended 


"Have  you  previously  attended:  TAC  ACES  I 

TAG  ACES  II  , NONE  (D-16) 

This  question  was  included  to  determine  the  extent  of 
the  subjects  experience  with  TAC  ACES  programs.  Specifi- 
cally, it  was  used  to  determine  if  any  relationship  exists 
between  the  performance  of  subjects,  with  any  or  no  TAC 
ACES  experience,  in  the  turkey  shoot  competition.  A total 
of  17  (19.1  percent)  of  the  89  subjects  in  the  study  respon- 
ded that  they  had  previously  participated  in  the  TAC  ACES  I 
or  TAC  ACES  II  training  program.  One  of  the  subjects  had 
completed  both  programs.  For  the  TAC  ACES  I program,  11 
respondents  in  the  sample  indicated  that  they  had  competed 
the  training.  When  contrasted  as  a group  with  the  total 
sample  of  turkey  shoot  participants,  it  was  found  that  the 
group  contained  one  turkey  shoot  winner  and  two  first 
runners-up  (second  place) . It  was  also  noted  that  none  of 
the  group  with  TAC • ACES  I training  had  finished  in  the  last 
quartile;  seventh  and  eighth  place.  Of  the  11  subjects  in 
this  group,  there  were  eight  subjects  (72.7  percent)  that 
finished  in  the  top  four  ranks  of  the  turkey  shoot  contest. 

The  mean  f-4  aircraft  flying  hours  experience  for  this  group 
was  333,6  hours. 

For  the  TAC  ACES  II  program,  seven  respondents  in  the 
sample  indicated  that  they  had  completed  the  training,  Of  the 
seven  subjects,  it  was  found  that  three  turkey  shoot  winners 
and  two  first  runners-up  (second  place)  were  in  this  relative- 
ly small  group.  One  subject  finished  in  the  last  quartile. 

It  was  also  found  that  six  subjects  (85.7  percent)  of  this 
group  finished  in  the  upper  three  ranks  of  the  turkey  shoot 
competition.  The  mean  F-4  aircraft  flying  hours  experience 
for  this  group  was  336.6  hours.  Further  analysis  indicates 


that  the  mean  Friday  GSI  score  Increased  by  28.1  percent 
for  the  group  with  prior  TAC  ACES  I experience.  The  mean 
Friday  GSI  score  increased  by  36.4  percent  for  rhe  group 
with  TAC  ACES  II  experience.  The  mean  Friday  GSI  score  in- 
creased by  38.7  percent  for  the  total  sample. 

Days  Since  Last  PACT 

"Date  of  last  Aggressor  DACT  Flight:  Less  than 

30  Days  , Less  than  180  Days  , More  Than 

180  days  , Never  . " (D-17) 

All  89  subjects  in  this  study  were  required  to  identify 
their  most  recent  Dissimilar  Aircraft  Training  (DACT)  ex- 
perience into  three  categories:  less  than  30  days,  less  than 
180  days,  and  more  than  180  days.  An  additional  category, 
"Never,"  was  provided  for  those  subjects  having  no  DACT 
experience.  Of  the  89  subjects,  their  DACT  experience  is 
distributed  as  roilows: 


less 

than 

30  days 

N = 10 

less 

than 

180  days 

N = 28 

more 

than 

180  days 

N = 14 

Never 

N = 37 

The  relationship  of  recent  DACT  experience  and  actual 
turkey  shoot  performance  is  contrasted  in  Table  38.  It  can 
be  seen  that  40  percent  of  those  subjects  with  the  most  recent 
DACT  experience  ( < 30  days)  were  also  winners  of  the  tur- 
key shoot  competition.  In  addition,  these  same  subjects 
(N  = 4)  comprised  one-third  of  the  total  group  of  12  turkey 
shoot  winners  in  the  study.  The  table  also  shows  that  more 
than  half  of  12  winners  had  some  DACT  experience. 

Six  of  the  10  subjects  in  the  first  category  ( < 30 
days)  were  either  turkey  shoot  winners  or  runners-up.  This 
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TABLE  38  - SUBJECTS  PER  CATEGORY 


DAYS 

SINCE  LAST  DACT  FLIGHT 

< 30  DAYS 

<180  DAYS 

>180  DAYS 

NEVER 

WINNERS 

4 

2 

1 

5 

RUNNERS-UP 

2 

4 

2 

4 

THIRD  ELIMINATORS 

3 

10 

3 

8 

SECOND  ELIMINATORS 

.1 

8 

3 

10 

FIRST  ELIMINATORS 

0 

4 

5 

10 

TOTAL 

10 

28 

L4 

37 

TOTAL  SAMPLE 


N = 89 


can  be  contrasted  with  the  winners  and  first  runners-up  in 
the  no  experience  - (Never)  category.  In  this  group,  only 
nine  subjects  (24  percent)  of  the  37  subjects  were  turkey 
shoot  winners  or  runners-up. 

Other  Visual  Air-To-Air  Simulators  Flown 


"What  other  visual  A/A  simulators  have  you  flown?" 

(D-18) 

The  question  was  included  to  determine  the  extent  of 
the  subject's  experience  with  other  visual  air-to-air  simu- 
lators. As  anticipated,  the  seven  subjects  that  responded 
to  the  question  concerning  TAC  ACES  II  experience  (D-16) 
also  responded  here,  and  they  were  deleted  from  this  analy- 
sis. A total  of  11  respondents  indicated  that  they  had 
flown  one  familiarization  flight  of  up  to  60  minutes  dura- 
tion in  the  TAC  simulator  for  air-to-air  combat  (SAAC) , Of 
this  group,  light  of  the  subjects  (72.7  percent)  had  a mean 
F-4  aircraft  flight  hours  experience  of  76.3  hours  and  three 
subjects  had  a mean  of  468.3  hours.  When  this  group  was 
contrasted  with  the  total  sample  of  turkey  shoot  partici- 
pants, the  results  were  inconclusive.  Only  one-  of  the  group 
was  a turxey  shoot  winner.  None  were  first  runners-up.  It 
was  also  found  that  seven  subjects  (63,6  percent)  of  the 
group  performed  in  the  lowest  two  quartiles  of  the  sample. 

Combat  Experience 

"How  many  combat  sorties  have  you  flown?  (D-19)  sorties." 
"What  is  your  total  combat  flying  time?  ( D-20)  hours." 
"Number  of  kills?  (D-21) . " 

"Number  of  hits  recorded.  (p-22)  . 1,1 
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"Number  of  SAM  encounters.  (D-23) . " 

"Number  of  hostile  aircraft  engagements.  (p-24)  . 11 
"Number  of  hits  received.  (D-24) . " 

The  questions  on  combat  experience  were  developed  to 
determine  the  degree  of  relationship  between  these  factors 
and  turkey  shoot  performance.  Eight  of  the  12  TAC  ACES  I 
classes  responded  to  the  questions. 

There  were  18  respondents  to  this  series  of  questions. 
A total  of  17  respondents  had  indicated  fighter  or  attack- 
type  as  their  aircraft.  One  respondent  indicated  a recon- 
naissance-type (RC-135)  and  was  not  included  here.  As  a 
group,  the  17  subjects  had  a mean  combac  flying  time  of 
316.1  hours  and  a mean  of  137.2  combat  sorties.  The  group 
had  flown  12  different  aircraft  types  in  combat.  This  in- 
cluded six  fighter  type,  three  attack  type,  and  three  ob- 
servation type  aircraft.  Results  indicate  that  there  was 
one  turkey  shoot  winner  in  this  group  of  17  subjects.  The 
subject  indicated  720  combat  flying  hours  experience  in 
observation  (0-2,  OV-10)  aircraft.  It  was  found  that  three 
subjects  finished  as  first  runners-up,  and  four  subjects  of 
the  group  finished  in  last  place.  The  group  was  also  con- 
trasted with  the  predicted  rankings  of  the  CIPs  with  simi- 
lar results.  The  instructors  ranked  eight  subjects  in  the 
upper  half  of  the  turkey  shoot  and  nine  subjects  in  the 
lower  half  (four  ranks) . The  results  indicate  that,  for 
this  sample,  combat  experience  of  this  type  is  not  a major 
factor  in  predicting  turkey  shoot  performance. 

Own  Training  Evaluation 

"What  is  the  value  of  the  overall  training  provided 
in  this  course  to  yourself?  (D-27) . " 
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This  question  was  contained  in  the  TAC  ACES  Program 
Evaluation  and  Critique  (see  Appendix  B) . 

The  questionnaire  was  developed  essentially  as  an  end- 
of-course  critique  for  the  TAC  ACES  program.  It  consists 
primarily  of  bipolar  descriptive  and  acceptability  scales. 
Narrative  space  is  provided  for  observations  and  other  com- 
ments. It  was  included  in  the  study  to  obtain  the  subject.' s 
perceived  value  of  the  training  they  obtained.  These  data 
were  to  be  used  to  assess  the  relationship  between  the  sub- 
ject's own  training  evaluation  and  turkey  shoot  performance. 
The  results  from  the  total  sample  of  89  subjects  show  that 
87  subjects  (97.8  percent)  evaluated  the  overall  training 
as  having  a positive  effect,  and  only  two  of  the  subjects 
evaluated  the  training  as  having  no  effect  on  their  per- 
formance. In  addition,  76  of  the  87  subjects  evaluated  the 
training  as  having  a substantial  positive  effect  on  their 
performance.  Both  subjects  who  responded  that  the  training 
had  no  effect  on  their  performance  finished  in  the  lower 
half  of  the  turkey  shoot  rankings,  and  one  finished  in  last 
place.  The  results  of  the  correlation  analysis,  as  shown  in 
Table  37,  indicate  the  correlation  of  this  variable  with 
turkey  shoot  rank , fractional  wins,  instructor  pilot  rank, 
and  CSX  scores  for  Monday  and  Friday.  It  can  be  seen  that 
the  "R"  values  are  quite  low  indicating  a lack  of  relation- 
ship between  this  variable  and  the  five  dependent  variables 
cited. 
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V.  PSYCHOMETRIC  AND  EDUMETRIC  DATA  ANALYSIS 


| DISCUSSION 

i 

j Individual  and  group  performance  data  were  recorded  for 

| all  the  89  subjects  in  this  study.  The  mean  GSI  performance 

| scores  for  the  Monday  and  the  Friday  data  sessions  were  cal- 

| culated  and  plotted  for  each  of  the  12  classes  and  are 

f 

| shown  in  Figure  19.  For  these  data,  two  least  squares  linear 

; trend  lines  were  computed,  using  the  number  of  classes  and 

i 

the  class  mean  Monday  GSI  scores  and  the  class  mean  Friday 
GSI  scores.  These  trend  lines  were  constructed  using  the 
data  in  Table  39. 

Four  of  the  12  TAC  ACES  classes  in  this  study  were 
subjected  to  separate  analysis.  In  addition  to  the  normal 
TAC  ACES  Monday  and  Friday  data  collection  sessions,  GSI 
performance  data  were  recorded  on  Wednesday  of  the  training 
week.  This  yielded  three  sets  of  performance  data  for  each 
of  the  four  classes.  Scatter  diagrams,  linear  and  quadratic 
curves,  and  frequency  distributions  were  constructed. 

For  clarification,  edumetrics  is  defined  here  as  the 
measurement  of  an  individuals  gains  from  training  experi- 
ences by  the  quantitative  assessment  and  analysis  of  per- 
formance data,  to  include  individual  and  group  data.  Edu- 
metrics is  shown  to  be  concerned  with  measures  of  learning 
performance  in  contrast  to  psychometrics,  which  is  concerned 
with  the  measurement  of  individual  differences  (i.e.,  mea- 
sures of  individual  innate  abilities  and  traits) . 

Psychometric  Analysis 

The  results  of  the  individual  performance  scores  for 
each  of  the  subject  pilots  in  the  four-class  sample  are 
shown  by  class  group  in  Figure  20.  A total  of  81  data 
points  were  used  to  fit  linear  and  quadratic  least-square 


^ A*? 


TABLE  39  - CLASS  AVERAGE  GSI  DATA 


CLASS  783 

M**7 


lines  for  all  four  classes  in  the  sample.  These  are  shown 
in  Figure  21  (For  clarity  of  presentation,  the  individual 
subject  data  has  been  grouped  by  class) . When  compared  with 
Figure  20,  it  can  be  seen  that  both  the  linear  and  the 
quadratic  equations  developed  approximate  the  centroid  of 
the  mass  of  data  points  for  each  pilot. 

Class  7826,  as  shown  by  the  data  in  Figure  20,  consis- 
ted of  four  students,  which  is  half  the  ^ize  of  the  normal 
TAC  ACES  class.  These  individual  pilots  received  more  in- 
tense instruction  and  training  due  to  the  lower  student/in- 
structor ratio  and  the  greater  amount  of  simulator  use  time 
available.  The  individual  performance  improvement  as  the 
length  of  training  increases  is  clearly  apparent  in  Figure 
20. 

Doth  the  linear  and  quadratic  lines  fit  the  data  well. 
Objective  measures  of  these  fits  are  shown  in  the  edumetric 
analysis.  The  quadratic  curve  is  preferred  in  describing 
the  data  because  it  approximates  true  learning  rates,  which 
tend  to  be  non-linear  as  a function  of  time.  Here  it  speci- 
fically shows  a higher  rate  of  learning  during  the  early 
phases  of  training  and  a lower,  slower  rate  during  the  final 
training  phases. 

The  distribution  of  the  GSI  scores  by  day  of  training 
are  shown  characterized  by  normal  distributions  in  Figure  22. 
It  can  be  seen  that  the  mean  ( X ) GSI  scores  improved  with 
length  of  training. 

Table  40  indicates  that  the  standard  deviation  of  the 
scores  decreased  as  length  of  training  increased.  This 
would  indicate  the  effects  of  learning.  The  reduced  varia- 
bility in  the  Wednesday  and  the  Friday  Standard  Deviation 
values  suggests  that  the  subjects  were  using  their  experi- 
ences gained  during  the  first  2-1/2  days  of  training  and 
calibrating  their  performance  responses  to  the  expected 
and  anticipated  performance  of  the  canned  targets. 
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TABLE  40  - 

EDUMETRIC 

DATA  BASE 

CLASS 

PILOT 

Y ■ GS1  SCORE 

NO. 

NO. 

MON ( X=fl ) 

wed(x=2.5) 

FRI(X=4.5) 

1 

359 

583 

595 

2 

312 

628 

601 

7826 

■a 

266 

471 

589 

4 

125 

508 

547 

1 

309 

494 

499 

2 

393 

743 

549 

3 

304 

590 

552 

7828 

4 

210 

635 

794 

5 

531 

638 

447 

6 

234 

332 

562 

7 

304 

649 

570 

8 

199 

414 

494 

1 

393 

546 

487 

2 

68  V 

617 

851 

3 

391 

522 

739 

7829 

4 

553 

524 

751 

5 

247 

317 

531 

6 

368 

441 

527 

7 

577 

469 

716 

8 

364 

521 

581 

1 

550 

631 

681 

2 

264 

595 

571 

3 

553 

449 

566 

7831 

5 

187 

676 

515 

6 

145 

631 

616 

7 

414 

590 

690 

8 

529 

— - ■—  — 

568 

773 

361.778 


147.563 


MEAN 
STD. DEV 
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547.481 

101.993 


607.185 

105.093 


Figure  20  is  included  to  show  the  degree  of  individual 
change  in  performance  score  for  each  subject  in  this  sample 
over  the  4.5  day  training  week.  The  data  indicate  the 
individual  subjects  had  a mean  performance  score  (GSI) 
improvement  of  61.3  percent  for  the  27  subjects  in  the  sample. 

Edumetric  Analysis 

The  GSI  Wednesday  performance  data  collected  for  four 
of  the  12  clasres  in  addition  to  the  normally  scheduled 
recordings  on  Monday  and  Friday  are  provided  in  Table  40. 

The  method  of  analysis  was  to  fit  a straight  line  and  a 
quadratic  curve  through  the  data.  The  objective  was  to 
ascertain  the  general  trend  in  GSI  scores  as  a measure  of 
group  learning  rates  as  the  classes  progressed.  The  X- vari- 
able chosen  was  days  of  training  completed.  Each  student 
was  assumed  to  have  no  training,  i.e.,  X=0,  on  Monday  when 
the  first  GSI  scores  are  measured.  The  students  were 
assumed  to  have  received  2.5  days  of  training  (X  - 2.5}  by 
Wednesday  and  by  Friday  morning,  4.5  days  of  training  (X  = 
4.5).  The  Y-variable  used  was  GSI  score. 

Figure  21  shows  a scatter  diagram  of  the  GSI  scores 
versus  days  of  training  using  the  data  provided  in  Table  40. 
The  figure  also  shows  the  linear  and  quadratic  least  squares 
curves  fit  through  the  data.  Both  curves  can  be  seen  to  fit 
well  through  the  central  regions  of  the  data  for  each  day. 
Also,  each  shows  the  general  trend  of  GST  Score  increasing 
with  days  of  training.  The  scatter  diagram  also  shows  the 
wide  variation  in  scores  for  each  day  and  the  general  over- 
lap which  occurs  from  day  to  day.  This  broad  variation  and 
day  to  day  overlap  also  points  out  the  general  weakness  of 
the  predictive  ability  of  the  initial  GSI  Score. 

The  linear  versus  the  quadradic  curves  are  contrasted 
in  Table  41.  Here  the  actual  linear  and  quadratic  equations 
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are  shown  along  with  an  analysis  of  variance  table  for  the 

linear  regression  and  "variation  breakdown"  for  the 

quadratic  equation.  The  multiple  correlation  coefficients 

(R)  are  also  provided  as  well  as  coefficients  of  determina- 
2 

tion  (R  ) for  both  equations.  The  F-ratio  for  the  linear 
model  is  included  and  is  significant  at  the  99.9  percent 
level,  (F  99q(l,79)  = 11.68).  This  indicates  that  the  slope 
of  the  straight  line  is  significantly  greater  than  zero  and, 
thus,  that  GSI  Score  increases  at  an  average  rate  of  about 
55  points  per  day  of  training  over  the  4-1/2  days  of  train- 
ing. 

2 

The  calculation  of  R (the  coefficient  of  determination 
or  the  multiple  correlation  coefficient  squared)  is  a mea- 
sure of  t-he  proportion  of  total  variation  about  the  mean  of 

the  GSI  score  explained  by  the  regression  line.  Thus  the 

2 

straight  line  explains  about  42  percent  (R  = .419)  of  the 

variation  and  the  quadratic  equation  explains  about  44  per- 
2 

cent  (R  = .440)  of  the  variation  between  training  time  and 
improvement  in  GSI. 

A test  was  also  made  for  "lack  of  fit"  of  the  straight 
line  to  the  GSI  Scores.  The  test  involves  breaking  the 
residual  sum  of  squares  into  two  parts,  one  part  measuring 
pure  error  and  the  other  measuring  lack-of-fit.  Repeating 
the  residual  sum  of  sciuares  for  the  straight  line  in  Table 

7 

41  results  in  the  following  breakdown: 

SOURCE  OF 


VARIATION 

D.F. 

SUM  OF  SQUARES 

MEAN  SQUARE 

F RATIO 

Residual 

79 

1,167,456.189 

Lack-of-Fit 

1 

43,686.708 

43,686,708 

3.032,262 

Pure  Error 

78 

1,123,769.481 

14,407.301,04 

F.95(1'78)  = 

3.92 

7 

Draper  & Smith.  Applied  regression  analysis.  New  York: 
John  Wiley  and  Sons , 1966',  26-31. 
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Now  since  3.032  < F g,~(lf78)  = 3.92  there  is  no  reason  to 
doubt  the  adequacy  of  the  linear  model,  i.e.,  the  lack  of 
fit  is  not  significant. 

A further  point  of  interest  is  the  actual  normality  of 
the  distributions  of  the  GSI  scores  being  analyzed  by  day, 
that  is,  is  there  any  reason  to  doubt  that  a given  set  of 
scores  is  normally  distributed?  The  Kolmogorov-Smirnov 
(K-S)  test  of  goodness  of  fit  was  applied  to  GSI  scores 

g 

for  each  day.  The  scores  were  found  to  be  normally  dis- 
tributed at  the  percent  significance  level  for  each  of  the 
three  sets  of  GSI  scores. 

Since  it  has  been  established  that  there  is  no  reason  at 
the  99  percent  level  to  doubt  that  the  GSI  scores  are  nor- 
mally distributed,  it  is  reasonable  to  present  Figure  23 
which  shows  three  normal  densities  with  parameters  (means  and 
standard  deviations)  equal  to  their  estimates  calculated  from 
the  GSI  scores  for  each  day.  This  figure  graphically  shows 
the  changes  in  GSI  Score  distributions  which  take  place 
during  the  course  of  training.  The  means  of  the  distribu- 
tions increase  with  training  time,  on  Monday  the  standard 
deviation  of  GSI  scores  is  compared  to  Wednesday  and  Friday 
(S(Monday)  = 147.6).  By  Wednesday,  however,  this  has  de- 
creased about  31  percent  over  Monday  (S (Wednesday) =102. 0)  and 
then  by  Friday  there  appears  to  be  a slight  increase,  (S (Fri- 
day) = 105.1).  To  determine  statistically  if  these  dif- 
ferences in  variance  exist,  Bartletts  chi-square  test?  for 
equality  of  standard  deviations  from  normal  distributions 

was  applied.  It  was  determined  that  the  null  hypothesis  of 

2 2 = 

no  difference  between  variances,  (H  : a (MON)  = o (WED) 

2 ° 
o ' (FRI) , cannot  be  rejected  at  the  95  percent  confidence 

level  but  can  be  rejected  at  the  90  percent  confidence  level. 


o 

Ostle  & Mensing.  Statistics  in  research  (3rd  ed.). 
Ames:  Iowa  State  University  Press,  197?) , 489-490. 

9 

Ostle  and  Mensing.  Statistics  in  research  (3rd  ed.). 
Ames:  Iowa  State  University  Press,  1975,  127. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


General 

An  experimental  investigation  has  been  performed  that 
statistically  validates  the  ability  of  an  empirically  de- 
rived performance  measure,  the  GSI,  to  correctly  predict 
student  pilot  performances  in  TAC  ACES  I free  engagement 
exercises.  The  empirically  derived  GSI  is  shown  to  exhibit 
correct  prediction  capabilities  of  student  pilot  performance 
comparable  to  that  of  expert  opinion,  subjective  student 
performance  predictions  by  instructor  pilots. 

The  empirically  derived  GSI  predictor  was  improved 
using  statistical  methods.  The  four  parameters  of  the  ini- 
tial (empirical)  GSI,  when  optionally  weighted,  were  shown 
to  predict  student  pilot  placement  in  the  turkey  shoot  with 
about  75  percent  accuracy.  These  four  parameters,  time  in 
gun  firing  envelope,  average  mil  error,  offensive/defensive 
time,  and  time  to  first  kill,  are  intuitive  to  the  experi- 
enced combat  pilot  as  measures  of  ACM  skill.  Each  of  the 
four,  when  objectively  measured,  can  be  used  as  teaching  aids 
in  the  development  of  air  combat  skill  in  the  student  pilot. 

Further  improvement  in  the  GSI  was  obtained  by  includ- 
ing certain  available  objective  and  subjective  parameters. 

The  optimal  methods  are  shown  to  be  excellent  predictors  of 
student  performance  (at  least  within  the  experiment  data) 
showing  probability  of  correct  student  performance  predic- 
tion near  80  percent  in  free  engagement  exercises. 

It  is  specifically  recommended  that  the  GSI  algorithms 
and  methodologies  of  this  initial  study  be  tested  in  the 
Simulator  for  Air- to- Air  combat  (SAAC)  at  Luke  AFB  and  on  the 
Air  Combat  Maneuvering  Instrumentation  (ACMI)  Range  at  Nellis 
AFB  to  determine  an  objective  measure  of  transfer  of  ACM 
training  between  the  simulator  and  the  aircraft. 
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Increased  Sample  Size 

The  results  of  the  study  yield  GSI  models  that  may  be 
applied  to  the  TAC  ACES  I population.  The  sample  size  used 
to  derive  these  models  was  relatively  small  (12  classes)  but 
was  related  to  the  whole  by  statistical  inference.  It  is 
desirable  to  continue  data  collection  and  statistical  analy- 
ses under  the  same  control  conditions  as  the  experiment  to 
accumulate  a larger  data  sample. 

It  would  be  useful  to  collect  additional  TAC  ACES  I 
data  for  the  following  reasons: 

1.  To  provide  a larger  sample  which  would  provide 
more  precise  information  on  the  distributions  of  the  data 
being  considered; 

2.  To  validate  the  predictor  models  derived  in  this 
study.  Careful  examination  of  GSI  data  collected  previous 
to  this  study  was  found  to  be  poorly  documented  and  of 
limited  use  in  validating  the  predictor  models.  Care  must 
be  taken  to  assure  that  reasonable  controls  are  placed  on 
the  data  collection  itself  as  lack  of  controls  affect  the 
validity  of  the  samples  themselves.  By  its  very  nature, 
this  kind  of  data  is  very  sensitive.  Lack  of  careful  sampl- 
ing can  result  in  collection  of  data  from  essentially 
different  populations  than  that  desired  and,  hence,  valida- 
tion becomes  difficult. 

Demographic  Data  Correlations 

The  master  data  base  provides  a means  for  further 
statistical  analyses  which  can  be  of  value  in  assessing 
training  and  training  requirements  in  ACM  simulators. 11 

^On  file  at  Vouyht  Corporation,  Dallas,  Texas. 


124 


It  is  recommended  that  ah  investigation  be  initiated  to  as*? 
certain  what  demographic  correlations  can  contribute  to  the 
overall  readiness  training  program.  In  order,  to  accomplish 
this)  objective,  it  is  necessary  to  continue  to  a)  collect 
these  data,  b):  supplement  these  data  with  other  data  which 
may  be  of-  value,  and  c)  analyze  the  data.!  to  obtain  correla- 
tion with  simulator  performance  measures*  and,  ultimately, 
d)  assess  performance  on  the  ACMI  range  exercises. 

Apply  GSI  to  Other  ACM  Simulator  Training 

The  parameters  comprising  the  GSI,  i;f  measured  in.  a 
similar  manner  and  under  similar  conditions,  are  applies-, 
hie  to,  other  A/2M-  simulator  .trainina,.  Thd  interrelation- 
ship of  these  parameters,,  i.e,,  weighting)  and  interaction, 
is  believed  to  be  specific  for  a particular  simulator  and 
training  syllabus.  It  is  recommended  that  the  GSI,  as 
derived  for  TAC  ACES  I,  be  introduced  as  a prospective 
measure  of  student  pilot  performance  in  a;h  ACM  simulator 
such  as  the  SAAC  and  adjustments  made  in  the  parametric 
contributors  to  develop  a statistically  derived  GSI  specific 
to  that  facility  and  training  syllabus.. 

The  GSI  Application  to  ACMI  Range 

The  .promise  of  the  GSI  as  a screening  tool  to  aid  in 
the  selection  of  fighter  talent  is  premature,  but  given 
a larger  data  sample  and  successful  application  of  the  GSI 
to  range  operational  exercises  such  as  the  ACMI  range  at 
Nellis  ■,  the- GSI  could  become  that  powerful!  tool. 


Potential  Utility  of  the  GSI 


The  GSI  was  shown  to  be  a measure  of  student  pilot  per- 
formance in  the  TAC  ACES  I Program.  GSI  scores  indicate 
the  relative  performance  of  students  in  the  simulator  and 
careful  scrutiny  of  the  GSI  contributory  parameters  can 
evaluate  the  strong  and  weak  points  of  a given  student  rela- 
tive to  his  overall  performance  measure.  These  "part  scores" 
are  associated  with  basic  flying  maneuvers,  tracking,  wea- 
pon switchology,  etc.  from  which  judgements  may  be  made  by 
the  instructor  pilot  where  to  concentrate  his  training 
efforts. 

The  GSI  may  also  be  utilized  to  obtain  a measure  of 
student  pilot  learning  trends  during  the  simulator  train- 
ing period.  The  skills  of  pilots  in  air  combat  can  vary 
greatly  depending  upon  individual  background  experience 
and  innate  ability.  The  individual  learning  abilities 
also  vary.  The  GSI  may  be  used  as  an  indicator  of  a 
pilot's  current  proficiency  in  air  combat,  as  well  as  an 
indicator  of  improvements  in  air  combat  skills  in  the 
simulator. 

The  GSI  can  be  used  to  establish  an  optimal  training 
period  for  the  norm  student  by  statistical  investigation  of 
initial  student  skill  and  skill  growth  over  training 
periods  varying  in  duration.  A cursory  survey  of  the 
12  class  sample  in  this  experiment  indicates  that  an 
optimal  training  period  in  the  simulator  can  be  established 
for  the  TAC  ACES  I population  by  further  statistical  analy- 
ses of  student  entry  skills  and  student  learning  trends. 

Contributing  parameters  that  comprise  the  Air  Combat 
Simulator  GSI  have  rudimentary  commonalities  with  many  other 
flight  simulator  training  devices.  It  is  probable  that 
other  flight  simulators,  i.e.,  Weapons  System  Trainers 
(WST) , Operational  Flight  Trainers  (OFT) , Instrument 
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Flight  Trainers  (IFT) , etc.,  can  utilise  the  same  or  simi- 
lar methodologies  as  presented  in  the  report  to  achieve 
comparable  simulator  performance  measures. 

Ucility  of  Data  Taken  During  Turkey  Shoot 

The  turkey  shoot  data  were  examined  to  investigate  the 
utility  of  the  data  collected  during  turkey  shoot  competi- 
tion. The  performance  measures  and  the  data  formats  were 
essentially  identical  to  those  used  in  the  GSI  data.  A 
basic  difference  is  that  performance  data  were  recorded 
separately  and  simultaneously  for  each  pair  of  combatants. 

No  GSI  scores  were  computed  from  this  data  set. 

The  performance  results  were  examined  for  a class 
selected  at  random. 

The  data  indicated  that  pilots  who  finished  in  the 
upper  half  of  the  turkey  shoot  had,  as  a group,  lower  mean 
minimum  altitude  values  than  pilots  who  finished  in  the 
lower  half  of  the  turkey  shoot.  The  data  snow  that  a suffi- 
cient body  of  pilot  performance  data  has  been  collected  to 
warrant  a detailed  statistical  analysis.  A cursory  examina- 
tion of  the  data  indicates  that  trends  of  a relationship 
appears  to  exist  between  turkey  shoot  rank  and  factors  such 
as  maximum  g,  minimum  altitude,  and  offensive  time.  The 
free  engagement  data  may  be  of  value  since  they  approximate 
engagements  on  an  air  combat  maneuvering'  range.  The  data 
may  also  be  useful  in  determining  links  between  GSI  per- 
formance predictors  and  those  predictors  to  be  determined 
for  the  ACMI  range(s). 


Limitations  of  the  GSI  System 


The  GSI  as  presented  in  this  report  is  specific  to 
TAC  ACES  1 training.  However,  its  application  to  other 
air  combat  simulator  training  where  the  environmental  train- 
ing features  are  similar,  i.e.,  training  hardware,  software, 
and  training  syllabuses  are  of  a similar  character,  may  be 
expected  to  yield  good  measures  of  air  combat  skill  (in  the 
simulator) . 

The  GSI  scoring  system  is  derived  for  air  combat  one- 
vers us-one  engagements  at  the  inception  of  of fensive/defen- 
sive  maneuvers.  In  its  present  form,  the  GSI  is  not  appli- 
cable where  initial  sighting  of  adversary  or  uwo-versus-one, 
or  one-versus-two,  is  instrumental  in  the  training  scenario. 

The  GSI  is  an  objective  indicator  of  air  combat  skill 
in  the  simulator  but  should  not  be  construed  as  an  absolute 
measure.  It  is  not  proposed  as  a substitute  for  subjective 
opinion.  When  the  two  measures,  GSI,  and  the  subjective 
opinion  of  the  instructor  pilot  are  u&eu  in  conjunction, 
they  produce  a maximal  evaluator  of  air  combat  simulator 
skill. 

GSI  Application  to  Other  ACM  Facilities 

The  degree  of  fidelity  of  simulation,  training  sylla- 
bus and  the  extent  of  training  are  factors  governing  trans- 
fer of  training  for  a given  task.  J.n  general,  ACM  simu- 
lator facilities  differ  widely  in  the  synergistic  fidelity 
of  air  combat. 

Lack  of  absolute  fidelity  in  a simulator  requires  the 
student  pilot  to  suppress  many  preconditioned  responses 
and  acquire  associated  responses  to  representative  exter- 
nal stimuli.  The  ability  of  the  student  to  transcend  to 
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this  representative  environment  directly  affects  his  per- 
formance in  a particular  simulator. 

The  differences  in  fidelity  of  simulation  between  simu- 
lators of  like  kind  and  the  difficulty  of  association  trans- 
fer experienced  by  the  student  will  determine  the  applica- 
bility of  the  GSI  to  other  ACM  simulators  as  a measure  of 
ACM  skill  and  as  a predictor  of  free  engagement  one-versus- 
one  contest  results. 

Some  examples  of  known  ACM  simulator  fidelity  differ- 
ences which  can  influence  GSI  application  are  motion/no- 
motion , g-suit/g-seat,  ground  rush  visual  cue,  and  the  ex- 
tent of  computer  modeling  of  aircraft  flight  characteristics 
(aerodynamic  fidelity,  control  response  fidelity,  instrument 
and  weapon  systems  fidelity) . The  effect  of  the  differences 
can  be  positive,  negative,  or  neutral,  on  the  contributory 
parameters  of  the  GSI. 
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APPENDIX  A 

ANALYTICAL  METHODOLOGY 


APPENDIX  A - ANALYTICAL  METHODOLOGY 


The  analytical  methodology  used  in  the  study  began  with 
preparation  of  elementary  statistical  displays  of  the  GSI 
and  the  four  component  variables  used  to  calculate  the  GSI 
score.  These  displays  consisted  of  histograms  and  scatter 
diagrams,  Variance-covariance  matrices  and  correlation 
matrices  were  also  generated  to  analyze  relationships  be- 
tween the  variables. 

Regression  analysis  was  used  extensively  in  an  attempt 
to  define  suitable  functional  predictive  relationships  be- 
tween the  various  candidate  predictor  variables  and  turkey 
shoot  outcomes. 

Two  Y-var..afcies  (dependent  variables)  were  considered 
in  the  regression  analysis.  They  were  turkey  shoot  rank, 
i.e.,  1,2, 3,4  ...,  and  fractional  wins.  "Fractional  wins"  is 
defined  as  the  ratio  of  total  wins  to  total  engagements  in 
the  turkey  shoot  for  a given  participant. 

Both  variable  selection  and  ridge  regression  were  used 
in  addition  to  all-variable  regressions  to  explore  the 
utility  of  direct  predictive  relationships.  Various  non- 
linear relationships  (in  the  candidate  predictor  variables) 
were  explored,  but  none  provided  relationships  as  good  as  a 
simple  GSI  ranking  predictor.  There  are  several  possible 
reasons  why  this  was  so:  Exploration  of  the  X'X  matrices 
indicated  that  in  all  cases  minimum  eigenvalues  were  very 
close  to  zero.  This  is  indicative  of  the  existence  of 
multicolinearities  in  the  predictive  variable  sets.  This 
condition  indicates  that  basic  assumptions  generall'  used  in 
the  application  of  least  squares  are  being  violated  and  also 
that  it  is  likely  the  parameter  estimates  will  vary  sub- 
stantially from  sample  to  sample.  Another  difficulty  was 
shown  to  exist  fzom  the  analyses  of  variance  performed. 

This  was  the  significant  variation  detected  between  classes. 
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The  regression  models  were  obviously  affected  by  this  and 
the  fact  that  no  constraints  were  (or  could  be)  applied  to 
rank  predictions.  For  example,  only  one  winner  is  allowed 
per  class,  but  several  might  be  predicted. 

In  general,  models  explored  using  ridge  regression 
showed  a degeneration  in  predictive  capability  as  the  bias 
factor  was  increased.  While,  in  general,  the  parameters  did 
stabilize,  as  might  be  expected,  the  predictive  rates  de- 
clined and  remained  unacceptable. 

The  all- variable , variable  selection,  and  ridge  regres- 
sion programs  used  in  this  study  had  been  developed  by 
Vought  previous,  to  the  beginning  of  this  study. 

As  it  became  apparent  that  the  regression  programs  were 
not  providing  useful  indicators  of  predictive  ability,  it 
was  decided  to  explore  three  sources  of  variation  in  the  GSI 
scores  and  the  GSI  component  variables.  Using  basic  analy- 
sis of  variance  methodology,  the  sources  of  variation  inclu- 
ded in  the  three-way  analysis  were  "between"  days,  "between" 
classes,  and  "between"  turkey  shoot  ranks.  In  general, 
significant  differences  tended  to  appear  between  days  and 
between  classes. 

At  the  beginning  of  the  study,  a master  data  base  was 
designed  and  then  implemented.  This  brought  data  from  the 
source  data  tapes  into  a common  file  where  it  could  be  con- 
veniently studied,  manipulated,  and  reduced  to  forms  suita- 
ble for  use  with  the  statistical  programs. 

The  next  ana  final  statistical  program  exercised  against 
the  data  was  the  Disc -iminant  Analysis  program  provided  in 
the  SPSS  package  available  on  Vought 's  System  370,  Discri- 
minant analysis  can  be  used  to  classify  data  sets  into  pre- 
defined groups.  In  the  case  of  this-  analysis,  the  groups 
were  defined  as  combinations  of  turkey  shoot  ranks.  As 


explained  in  the  main  body  of  the  text,  this  part  of  the 
analysis  was  performed  for  four  different  group  defini- 
tions with  four  different  data  sets.  The  program  was 
always  operated  in  the  variable  selection  mode  using  the 
Lambda  variable  selection  option.  Data  sets,  prior  to  in- 
put, were  sorted  by  turkey  shoot  rank  with  all  winners  at 
the  top  of  the  list,  runners-up  following,  and  so  on.  Pro- 
gram control  parameters  were  then  used  to  define  the  number 
of  groups  and  the  number  of  members  of  each  group.  As  noted 
above,  four  groupings  were  defined  for  four  different  data 
sets.  Thus,  in  all,  16  discriminant  analyses  were  per- 
formed. These,  in  general,  provided  the  best  predictors  of 
turkey  shoot  outcomes  developed  in  the  study.  The  results 
are  documented  in  the  main  body  of  the  report. 

Several  other  commonly  used  statistical  techniques 
were  also  employed.  Among  these  were  the  calculation  of 
confidence  intervals  on  the  proportions  of  correct  classi- 
fications of  cases  by  the  discriminant  program  using  data 
from  the  12-class  sample.  This  procedure  made  use  of  the 
normal  approximation  to  the  binomial  distribution  which 
is  often  used  where  sample  size  is  adequate.  Certain  tests 
of  hypotheses  were  also  used  during  the  comparison  of  the 
discriminant  results  calculated  from  the  12-class  sample 
and  with  the  four  classes  of  data  used  to  test  various  pre- 
dictors. This  was  used  to  test  equality  of  prediction  rates 
of  the  discriminant  predictors  on  the  12-class  experiment 
data  with  the  four  class  test  data. 

Certain  other  tests  were  employed  to  test  for  normality 
of  data  and  applicability  of  a straight  line  to  the  learn- 
ing rate  data  used  in  the  edumetric  analysis.  Footnotes 
are  used  in  the  main  text  to  identify  references  applicable 
to  the  statistical  methods  employed. 


APPENDIX  B 

FORMS  UTILIZED  IN  THE 
GOOD  STICK  INDEX  VALIDATION  STUDY 


TAG  ACES  BACKGROUND  SURVEY 


1.  FULL  NAME RANK  (D-l)  2.  DATE 

3.  CLASS  & PILOT  # 4.  ACES  I F],  ACES  II  Q 

5.  MIL  ADD.  SQDN  (D-2)  WING  ( D- 3 ) BASE  (D-0)  ZIP  (D-0) 

6.  TOTAL  FLYING  TIME  (D-4)  7.  TOTAL  FIGHTER  TIME  (D-5) 

8.  TOT.  F-4  TIME  (A/C  & IP) (D-6)  9.  SORTIES  (LAST  6 MOS) (D-7) 

10.  CURRENT  IN:  F-4C  Q , F-4DQ  , F-4EQ  , (OTHER)  (D-8)  Q 

11.  PRIMARY  DOC:  A/aQ  , A/G[]  , RTU  IPQ  , (OTHER)  (D-9)  □ 

12.  RECENT  BFM/ACM  EXPERIENCE:  SORTIES -TOTAL  (D-10) , 

LAST  6 MOS  (D-ll) , LAST  MO  ID-12) 

13.  TIME  SINCE  LAST  BFM/ACM : 0-2  WKSQ  , 3-4  WKSQ]  , 

5-12  WKsD  , 13-25  WKSD  , 26-52  WKSQ  (D-13) 

14.  WHAT  A/A  MISSILES  HAVE  YOU  TIRED?  AIM-7Q  , AIM-9Q  , 
AIM-4Q  , N0NEO  ( D-14) 

15.  ARE  YOU  AN  FWIC  GRADUATE?  YESQ  > NOfl)  (D-15) 

16.  HAVE  YOU  PREVIOUSLY  ATTENDED:  TAC  ACES  1^  , 

TAC  ACES  IlQ  , NOQ  (D-16) 

17.  DATE  OF  LAST  AGGRESSOR  DACT  FLIGHT:  LESS  THAN  30  DAYS[U  , 
LESS  THAN  180  DAYSG  , MORE  THAN  180  DAYSf^]  , 

NEVERQ  (D-17) 

18.  WHAT  OTHER  VISUAL  A/A  SIMULATORS  HAVE  YOU  FLOWN?  (D-18) 


19,  COMBAT  EXPERIENCE:  YES  Q,  NO  □.  IF  YOU  HAVE  HAD  COMBAT 
EXPERIENCE,  WHAT  IS  YOUR  TOTAL  COMBAT  FLYING  TIME?  (D-20) 
HOURS.  HOW  MANY  COMBAT  SORTIES  HAVE  YOU  FLOWN?  (D-19) 
SORTIES.  WHAT  TYPE  OF  AIRCRAFT  HAVE  YOU  FLOWN  IN  COMBAT? 
(D-0)  NO.  OF  AIRCRAFT  ENGAGEMENTS  (D-24) . NO.  OF  HITS 
RECORDED  (D-22) . NUMBER  OF  HITS  RECEIVED  (D-25) . NUMBER 
OF  KILLS  ( D-21) . NUMBER  OF  SAM  ENCOUNTERS  (D-23) . 

20.  DATE  OF  BIRTH  (D-0) . 

*D-0  - NOT  INCLUDED  IN  ANALYSIS 
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TURKEY  SHOOT  FORMAT 

DOUBLE  ELIMINATION 


2ND  ELIM 


INSTRUCTOR  OPINION  FORM 


In  your  opinion,  how  will  each  of  the  students  in  class  

perform  in  the  Turkey  Shoot  Competition?  Please  rank-order  the  students 
on  a scale  of  from  1 to  8.  Use  the  rank  of  1 to  identify  the  student  who 
you  feel  will  win  the  Turkey  Shoot,  th-  ra.-k  of  2 to  identify  the  first 
runner-up,  and  so  on  until  the  rank  ol  8 to  identify  the  student  who  you 
feel  will  place  last.  Please  rank  all  the  students. 


INSTRUCTOR  PILOT  DATE 


NOTE:  Please  complete  this  form  before  the  student  Turkey  Shoot  Competition 

each  Friday.  lbs  form  will  be  collected  from  you  by  Mr.  R.A,  Jor^e  sen. 
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NOTE:  COMPLETED  FORM  WILL  BE  COLLECTED  BY  

MR.  R.  A.  JORGENSEN.  REAL  TIME  COMPUTING 


TAC  ACES  PROGRAM  EVALUATION  AND  CRITIQUE 


NAME/RAHK CLASS  # PILOT  # DATE 

TAC  ACES  PROGRAM:  I ( ),  II  ( ) 

BOTE:  This  evaluation  will  he  conducted  in  three  parts. 

In  part  I you  are  asked  to  give  your  ratings  of  the  utility  of  this  training 
concent.  In  short,  would  regular  exposure  to  visual  air-to-air  simulation  he 
beneficial?  Does  it  possess  the  potential  to  increase  your  combat  capability? 

In  part  II  you  are  asked  to  assess  end  rate  the  relative  benefit  of  the 
simulator  itself,  including  instructional  features.  What  improvements  must  it  have? 
Where  is  it  good  enough? 

Part  III  consists  of  unstructured  questions  relating  to  simulator  training 
capabilities  and  limitations,  course  value,  instruction,  and  tho  TAC  ACES  program 
in  total. 


PART  Is 

Use  the  following  scale  to  rate  each  question  end  add  appropriate  comments 
whan  necessary: 


Rating  General  Meaning 

5 Subatsnticl  positive  training 

A Slight  positive  training 

3 No  effect 

2 Possible  negative  training 

1 Definite  negative  t reining 


A.  What  in  the  value  of  the  overall  training  provide:?  in  this  course  to: 


5 

4 

* 

*• 

C 

1 

COMMENT 

Experienced  pilots 

- - - . . _ - _ 

Inexperienced  pilots 

Yourself 

A/A  DOC  pilots 

A/0  DOC  pilots 

RTU  IFa 

■PART  II 


The  sinulator’ s chief  purpose  is  to  aid  the  instructor  in  teaching  various 
air-to-air  tasks.  As  such  it  should  be  like  the  aircraft  in  many  respects  but  not 
necessarily  in  every  detail.  In  addition  i'  should  be  design  d to  ease  the  workload 
on  the  instructor  while  still  providing  effective  control  over  the  engagement. 

A.  Crap are  the  sinulator  to  the  aircraft  in  she  following  areas  using  the  rating 
scale  provided: 

5 - Much  better  than  aircraft 
4 - Slightly  better 
3 - About  tho  sane 
2 - Slightly  worse 

1 - Much  worse  — — — — 

5 4 j 3 j 2 l cotes;  t 

Acceleration  Performance  j I 


Deceleration  Performance 
Roll  Performance 
Pitch  Performance 


Yaw  Performance 


Turn  Rate 

AOA  Indications 
(buffet,  tone,  noise) 

Longitudinal  Stick  Feel 
Lateral  Stick  Feci 
Rudder  Feel 
AIK-7  Performance 
AIM-9  Performance 


Gun  Performance 


Gunsight  Performance 
IR  Tone  Operation 


B.  Do  you  feci  cockpit  notion  is  necessary  for  an  A/A  simulator?  Yes  ( ) No  ( ) 

Consent: 


PART  III 

A.  What  A/A  tasks  and/or  RrM  maneuvers  CAN  bo  trained  in  the  simulator? 


B.  What  do  you  consider  to  be  the  best  training  features  of  this  simulator? 


C,  What  A/A  tasks  and/or  B7M  maneuvers  CARROT  be  trained  i.«  the  simulator? 
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D.  What  do  you  consider  to  he  the  most  significant  limitations  of  this  simulator? 


E.  Has  the  training  provided  during  this  week  improved  your  overall  operational 
fighter  skills?  Yes  ( ) Ho  ( ) 

Comment: 


F.  Should  the  course  be  offered  on  a recurr.'ng  basis?  Yes  ( ) Ho  ( ) 

Comment: 


G.  Comment  on  the  quality  and  quantity  of  instruction. 


H.  What  features/capabilities  would  you  like  to  see  added  to  this  simulator? 


1.  Hat  any  coinents/reconsnendations  you  have  regarding  the  TAC  ACES  program, 
{i.e. , syllabus/administrati ve/scheduling/quarters/transportation/etc. . . ) 
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APPENDIX  C 


TAG  ACES  I TRAINING  SYLLABUS 
AND 

TURKEY  SHOOT  COMPETITION  RULES 


TAC  ACES  I 
SYLLABUS 


DAY  1 


Sortie  #1  - Simulator  Familiarization  (:30)  (F-4/F-4) 

Objective:  To  become  familiar  with  simulator  visual 
display,  switchology,  aural  and  dynamic  cues,  flight 
controls,  and  performance  characteristics. 

Pilot  will  perform  following  tasks: 

a.  Acceleration  maneuvers 

b.  Rolling  maneuvers 

c.  Turning  maneuvers 

d.  High  and  low  altitude  flight 

e.  High  and  low  speed  stalls 

Sortie  #2  - Weapons  Familiarization  ( : 30 ) (F-4/F-4) 

Objective:  To  become  familiar  with  AIM-7E,  AIM-9J, 
and  20mm  employment. 

Pilot  will  perform/demonstrate  following  tasks: 

a.  AIM-7  and  AIM-9  employment  against  a con- 
trolled target 

b.  Gun  tracking  exercises  against  a controlled 
target 

c.  Understanding  of  weapons  switchology 

d.  Recognition  of  aspect  angle,  range,  and 
closure  velocity, 

e.  Max  performance  maneuvering 

Sortie  13  - Performance  Measurement  Data  (:30)  (F-4/ 
Computer  Flown  Target) 

Objective:  To  collect  a baseline  performance  mea- 
surement on  each  pilot  as  he  flys  against  a pre- 
recorded profile. 

The  performance  measurement  will  consist  of  the 
following  exercises: 

a.  2 x Stabilized  (Cine)  tracking  exercises 

b.  3 x Head-On  maneuvering  exercises 
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DAY  2 


Sortie  #4  - Gun/Tracking  (1:00)  (F-4/F-4) 

Objective:  To  fully  understand  operation  and  em- 
ployment of  gun  and  LCOSS. 

Each  pilot  will  accomplish: 

a.  Stabilized  tracking  exercises 

b.  High  angle  gun  employment 

c.  Tracking  a maneuvering  target 

Sortie  #5  - Basic  Fighter  Maneuvers  - Offensive  (1:00) 
(F-4/F-4) 

Objective:  To  understand  and  be  able  to  perform 
basic  fighter  maneuvers  from  a canned  set-up. 

Each  pilot  will  perform  the  following: 

a.  High  and  low  Yo-Yo 

b.  Quarter  plane  maneuver 

c.  Lag  roll 

d.  Acceleration  and  separation  maneuvers 

DAY  3 

Sortie  #6  - Basic  Fighter  Maneuvers  - Defensive  (1:00) 
(F-4/F-4) 

\ 

Objective:  To  understand  energy  management  and  basic 
defensive  maneuvers. 

Each  pilot  will  understand  and  practice: 

a.  Overshoots 

b.  Extensions 

c.  Reversals 

d.  Jink-outs 

Sortie  #7  - Air  Combat  Maneuvering  - Similar  (1:00) 
(F-4/F-4) 

Objective:  To  increase  proficiency  in  entire 
maneuvering  envelope. 
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Each  pilot  will  demonstrate  understanding  of: 


a. 

Use  of  the  vertical 

b . 

Lead  turn 

c. 

High  AOA  maneuvering 

d. 

Combat  separations 

DAY  4 

Sortie 

#8  - 

Threat  Orientation  (1:00)  (F-4/Threat) 

Objective:  To  develop  an  appreciation  for  the  per- 
formance characteristics  of  a typical  threat  air- 
craft. 

Each  pilot  will  observe  the  following  threat  charac- 
teristics : 

a.  Flight  control  responses 

b.  Turning  capability 

c.  Performance  envelope  (altitude,  airspeed, 
etc. ) 

Sortie  #9  - Air  Combat  Maneuvering  - Dissimilar  (1:00) 
(F-4/Threat) 

Objective:  To  increase  proficiency  in  maneuvering 
against  dissimilar  aircraft. 

Each  pilot  will  fly  each  aircraft  in  fluid  engage- 
ments against  each  other.  Lessons  learned  will  be 
discussed  during  debriefing. 

DAY  5 

Sortie  #10  - Review  of  Sorties  1-9  ( : 4 5 ) (F-4/F-4) 

Objective:  Briefly  review  all  previous  sorties  for 
areas  of  confusion/misunderstanding . 

Each  pilot  will  demonstrate  knowledge  of  basic  con- 
cepts of  air-to-air  combat  maneuvering. 

Sortie  #11  - Performance  Measurement  Data  ( : 15 ) (F-4/ 

Computer  Flown  Target) 

Objective:  To  collect  an  end  of  course  performance 
measurement  as  the  pilot  flys  against  a pre-recorded 
profile. 
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The  performance  measurement  will  consist  of  the 
following: 

a.  2 x Stabilized  (Cine)  tracking  exercises 

b,  3 x Head-On  maneuvering  exercises 

Sortie  #1°  - Turkey  Shoot  (F-4/F-4) 

Objective:  To  allow  pilots  to  demonstrate  their 

air-to-air  ability  in  a class  fly-off. 

Each  pilot  will  be  eliminated  after  losing  to  two 
other  pilots  in  a double  elimination  tournament. 
Rules  of  engagement  will  be  briefed  prior  to  start 
of  fly-off. 


— On  all  1.0  hour  sorties,  pilots  will  switch  cockpits 
after  first  30  minutes. 

— Sorties  should  be  recorded  for  debriefing. 
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TURKEY  SHOOT  RULES 


Double  Elimination 


1.  Initial  pairings  will  be  made  by  drawing  names  from 
a hat. 

2.  Both  aircraft  will  be  F-4E's  at  15,000  feet  and  425 
kts , head  on  at  18,000  slant  range. 

3.  Paired  participants  will  flip  a coin  for  choice  of 
cockpit. 

4.  There  will  be  a 3 minute  time  limit  for  each  engage- 
ment. After  3 minutes,  both  aircraft  will  be  reset 
to  the  initial  set-up. 

5.  Aircraft  over-G  (10  G's),  hitting  the  ground,  and 
spins  that  bomb  the  computer  are  automatic  kills. 

6.  Head  on  gun  kills  are  not  authorized.  An  aspect  angle 
greater  than  135  degrees  for  the  shooter  at  time  of 
kill  is  considered  a head  on  gun  kill. 

7.  Radar  lock-on  can  only  be  accomplished  by  pilot  acti- 
vated auto-acq  after  the  second  engagement.  Radar 
missiles  will  not  be  used  until  the  third  engage- 
ment. 

8.  Switchology  trickology  is  unauthorized. 

9.  Entry  fee  will  be  decided  by  the  class  (normally  $1/ 
pilot)  . 

10.  These  may  be  agreed  to  or  changed  by  the  entire  class. 

11.  Lie,  cheat,  and  steal,  but  keep  your  six  clear  and  may 
the  better  man  win l I 

12.  Head-on  kills  on  the  initial  pass  are  not  authorized 
at  any  time . 
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LIST  OF  ABBREVIATIONS,  ACRONYMS  AND  SYMBOLS 


ACES 


ACMI 


AFHRL 


CIPP 


CTK,  CT.NETK 
D,  DEM. 
DISCRIM 
DF 

Elirc. 


F - ratio 


F test 


FWIC 
G,  g 


- Air-to-Air 

- Aircraft 

- Air  Combat  Engagement  Simulator 

- Air  Combat  Maneuvering 

- Air  Combat  Maneuvering  Instrumentation 
Range 

- Air  Forces  Human  Resources  Laboratory 

- Average 

- Basic  flying  maneuvers 

- Chief  Instructor  Pilot 

- Chief  Instructor  Pilot  predictions  of 
turkey  shoot  ranking 

- Classification  Function 

- Cinetrack  exercise  in  tracking  maneuvers 

- Demographic  data 

- Discriminant  analysis  program  used 

- Degrees  of  freedom 

- Eliminated{ors)  from  Turkey  Shoot 

- Envelope 

- Expanded  {list  of  variables) 

- Friday  scoring  data 

- Function 

- Variance  between  groups  divided  by 
variance  within  groups 

- Test  of  significance  used  in  analysis 
of  variance 

- Flight  Training  Operations 

- Fighter  Weapons  Instructor  Course 

- Acceleration  relative  to  that  of  gravity 

- Greather  than 

- Group 

- Good  Stick  Index 
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LIST  OF  ABBREVIATIONS , ACRONYMS  AND  SYMBOLS  (Cont.) 


H 

H1 

H-MIS 

H0 

HON,  HD-ON 

INT.,  Int. 

IP 

LBS 

LT, 

M 

MIL  ERR 
N 

O/D 


OTOPS 

PANG 

PE 

Pred. 


R 

R2 

R-MIS 

R.U. 

S 


SAAC 

SAM 

SR 


- Hit 

- Hypothesis  where  P£  ^ PT 

- Heat  missile 

- Hypothesis  where  PE  = P^ 

- Head-on  exercise 

- Internal 

- Instructor  Pilot 

- Pounds  (fuel) 

- Less  than 

- Monday  scoring  data 

- Average  pointing  error  in  Mils 

- Sample  size 

- Ratio  of  offensive  time  (target  in  front 
hemisphere  of  subject  aircraft)  to  defen- 
sive time 

- Opaque  Target  Optical  Projector  System 

- Pointing  Angle  Advantage  (Time  in  envelope) 

- Proportion  of  correct  classifications 
using  data  within  the  experiment 

- Prediction (s)  ~or(s) 

- Proportion  of  correct  classifications 
using  test  data  from  outside  the  experi- 
ment 

- Correlation  coefficient 

- Coefficient  of  determination 

- Radar  missile 

- Runner (s) -Up  of  Turkey  Shoot 

- Standard  deviation 

- Simulator  for  Air-to-Air  Combat 

- Surface-to-Air  Missiles 

- Slant  range 
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LIST  OF  ABBREVIATIONS,  ACRONYMS  AND  SYMBOLS  (Cont.) 


1 


ss 

TAC 

TAG  ACES  I 
II 

TAS 
TFWC 
T.S . , TS 
TTFK 

VAR 

o2 

W 

Win. 

X 


Y 


- Sums  of  squares 

- Tactical  Air  Command 

- Simulator  training  program  at  Vought 

- at  Luke  Air  Force  Base 

- Training  and  simulation 

- Tactical  Fighter  Weapons  Center 

- Turkey  Shoot 

- Time  from  start  of  engagement  to  first 
kill 

- Variable 

- Variance 

- Wednesday  scoring  data 

- Winner (s)  of  Turkey  Shoot 

- Sample  mean 

- Chi-Square  test  statistic 

- Variable  quantities 

- Dependent  variable  quantity 


i 
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